39 Commits

Author SHA1 Message Date
André Koch-Kramer
91d5d5f867 Add class and functions for downloading highlights
Requested in #162.
2018-08-23 22:19:08 +02:00
Alexander Graf
0dcc912987 doc: Note new features with versionadded 2018-08-23 17:25:28 +02:00
Alexander Graf
e388a1c966 --no-posts; Deprecate --{profile-pic,stories}-only 2018-08-20 16:29:14 +02:00
Alexander Graf
06b7edd6d5 --no-pictures flag to not download post pictures
Closes #131.
2018-08-20 14:27:58 +02:00
Lars Lindqvist
e00d77c234 Basic CLI support for tagged posts.
Squashed commit of the following (pr #154):

commit 8fd56c379ff89ff634b510df8abde5a9c50218f0
Merge: 08f0ee7 a3777ca
Author: Lars Lindqvist <e5150@users.noreply.github.com>
Date:   Wed Aug 15 20:23:23 2018 +0200

    Merge branch 'master' into master

commit 08f0ee795c2273b09056031253781776c5c93bf4
Merge: 700b3a8 dcea0e9
Author: Lars Lindqvist <e5150@users.noreply.github.com>
Date:   Sun Aug 5 15:25:55 2018 +0200

    Merge branch 'master' into master

commit 700b3a8d094f552caa638a9d91f9221392a8e3f0
Author: Lars Lindqvist <lars.lindqvist@yandex.ru>
Date:   Sat Aug 4 16:26:59 2018 +0200

    Basic CLI support for tagged posts.

commit 5e3cd10cbcbec6d29abd6a56ab9c39294a8d44b3
Merge: af564f5 92653dc
Author: Lars Lindqvist <e5150@users.noreply.github.com>
Date:   Fri Aug 3 19:33:24 2018 +0200

    Merge branch 'master' into master

commit af564f5174d1ffd3af3f7e635b650651e1f7411a
Author: Lars Lindqvist <e5150@users.noreply.github.com>
Date:   Fri Aug 3 19:25:57 2018 +0200

    Fix owner_profile for Profile.get_tagged_posts()

commit 3cde1f7db4860edaca970c70543ed5bca0f97853
Author: Lars Lindqvist <e5150@users.noreply.github.com>
Date:   Thu Jul 26 19:51:33 2018 +0200

    Add meth:get_tagged_posts to Profile
2018-08-20 08:50:29 +02:00
Alexander Graf
a070ed42ae get_stories(): Fetch in chunks of 100 users
It seems there is a limit of how many user stories can be requested at once.
Now, Instaloader.get_stories() chunks the stories request.

Fixes #157 and fixes #134.
2018-08-05 08:06:53 +02:00
Lars Lindqvist
bfa5d8ed41
Fix :stories
50a5330fecb1272487fdd600416354c83ef8922f breaks `:stories` if userids is None (non-iterable).
2018-08-05 03:29:24 +02:00
Alexander Graf
7f81985911 _ArbitraryItemFormatter: Allow chaining attributes
Override get_value() rather than get_field(), to allow chaining of item
attributes, such as --post-metadata-txt={owner_profile.profile_pic_url}.

Motivated by #151.
2018-08-04 11:41:19 +02:00
André Koch-Kramer
50a5330fec Change download of stories when not using :stories
If using --stories or --stories-only the stories got donwloaded along
with the profiles one by one. Now, the stories get downloaded in a
similar aproach like when using the :stories target, i.e.
download_stories() gets only called once.
Profile.has_highlight_reels is broken and now always returns true.
This fixes #153.
2018-08-02 16:20:27 +02:00
Alexander Graf
2df1419d68 Fix file extension of profile picture
Fix #142.
2018-07-09 10:07:16 +02:00
Alexander Graf
86035b8d2d Fix obtaining csrftoken for login
Fixes #122. Further fixes processing of login JSON response regarding error
handling.
2018-05-23 08:50:37 +02:00
Alexander Graf
a3eb93c21a Profile: Cache iphone data structure if queried
Profile.get_profile_pic_url() and Profile.has_highlight_reels did the same
query without remembering the response, resulting in that query being made
twice when downloading stories and profile pic of a profile.
2018-05-19 09:52:07 +02:00
Alexander Graf
d19dd2c9d4 Fix internal references in docs/as-module
Further, document all attributes that are referred to, and ensure that there
will be no broken references again by letting Travis call Sphinx with -W -n.
2018-05-14 10:46:04 +02:00
Alexander Graf
52eb6d8e27 Fix minor issues regarding last commit
- Close session from InstaloaderContext.anonymous_copy().

- Have graphql_count_per_slidingwindow default value in one place rather than
  three.
2018-05-14 10:27:34 +02:00
André Koch-Kramer
3211d63ec1 Replace deleted field has_highlight_reel
Since the property has_highlight_reel is no longer available throught
the previously used graphql query, this information needs to be obtained
in another way. Therefore the properties has_highlight_reels,
has_public_story and has_viewable_story were added to the Profile class.
Since has_public_story can be obtained throught graphql queries without
being rate limited when invoked anonymously, the ability to use an
anonymous copy of the context was added to to the InstaloaderContext class.
Fixes #116
2018-05-13 19:43:49 +02:00
André Koch-Kramer
e1228501d2 Set graphql_count_per_slidingwindow to 20
Instagram tightened their rate limits again.
Implemented hidden switch --graphql-rate-limit for easier adjustment of
graphql_count_per_slidingwindow.
Fixes #118
2018-05-13 19:09:40 +02:00
Alexander Graf
211b2269ff download_profile: Request story only if has_highlight_reel
This is a simple modification to save GraphQL queries, hence to more likely
stay within rate limits without having to wait between profiles.

Fixes #112.
2018-05-08 18:10:28 +02:00
Alexander Graf
b3df5a5f07 Fix bugs in Instaloader.update_comments()
- Cast id and created_at from JSON to int to be compatible with comments JSON
  files from Instaloader 3.3,
- Do not fail if comment JSON files are empty,
- Close file descriptors of comment JSON files when reading,
- Do no create empty comment JSON files if generating content fails.

Closes #114.
2018-05-08 17:39:00 +02:00
Alexander Graf
b1e55a41a4 Do not log Post's username in download_saved_posts
This improves speed, as get_saved_posts() returns Posts that do not have
owner_profile info. So, printing the owner_username often required an
additional request.
2018-05-01 20:22:52 +02:00
Alexander Graf
084cfb102c Handle ValueError on bad profile id file
Fixes "ValueError: invalid literal for int() with base 10: b''", reported in #95.
2018-05-01 15:14:19 +02:00
Alexander Graf
2722da6ae4 Fix anon username from id (find renamed profile)
maybe related to #95, #104, 67ac8f339716c498cddb2f1489eac6104409646c.
2018-05-01 13:11:40 +02:00
Alexander Graf
8a8ea2913c Improve post/storyitem-metadata-txt behavior
If an expression evaluates to None, an empty string is yielded, rather
than 'None'.

Blanks (and newlines) are strip()ed from the formatted strings, which
also avoids creation of whitespace-only textfiles.

save_captions parameter to Instaloader constructor has been removed, as
it is equivalent to setting both {post,storyitem}_metadata_txt_pattern
to '' (empty string).

Instaloader.save_caption() now prints the '[{old}] updated [{new}]' if
the caption was updated, rather than printing the new caption twice,
which was confusing.
2018-04-29 11:17:48 +02:00
Alexander Graf
b2dd395a86 Link cli-options in Instaloader __init__ docstring 2018-04-29 10:13:31 +02:00
Alexander Graf
d21fb4154e fine-tune as-module.rst 2018-04-28 21:45:57 +02:00
Alexander Graf
147cd1a580 Post_get_comments() yield namedtuple PostComment 2018-04-28 18:17:00 +02:00
Alexander Graf
3dcfcff04c namedtuples PostLocation and PostSidecarNode
Post.get_sidecar_edges() has been renamed to Post.get_sidecar_nodes().

Post.get_location() has been made to a property Post.location and it now
internally stores location struct and makes it JSON store-/loadable.
2018-04-28 17:11:08 +02:00
Alexander Graf
2c50972e08 Update as-module.rst and docstrings 2018-04-28 15:47:04 +02:00
André Koch-Kramer
e25eb2a948 Limit GraphQl queries to 20 per 11 minutes
cherry-picked from commit d90c05e0a44cb0c41b98436ac18bfd1955a0a9ce
- Set GRAPHQL_PAGE_LENGTH to 50 what appears to be the new working maximum.
- Limit GQL queries to 20 per 666 seconds.
- Remove logic for tracking queries per query identifier as Instagram
  only allows 20 overall GQL queries per sliding window.

Related to #101
2018-04-26 17:13:59 +02:00
Alexander Graf
7d2830b8b2 Have --storyitem-filter (--only-if for StoryItems) 2018-04-19 15:12:05 +02:00
Alexander Graf
59152214cf Allow specification of what to write in txt files 2018-04-13 22:54:40 +02:00
Alexander Graf
80f701150c New formatter for filename pattern 2018-04-13 21:19:04 +02:00
Alexander Graf
df1cdb5d48 Remove now-unneeded Tristate 2018-04-13 19:07:23 +02:00
Alexander Graf
8ab7d3398f Rename Instaloader.download_story{,item}
It makes more sense since the object is called StoryItem.
2018-04-12 22:04:06 +02:00
Alexander Graf
f0bebd0d96 Saving and reimporting of JSON files
Metadata JSON files are now created by default. They can later be given
as a target to redownload Posts, StoryItems or Profiles with new
settings.
2018-04-12 22:03:02 +02:00
Alexander Graf
9b3014d5bf Fix anonymous GraphQL queries
Port of 73ec884ea49cf099570b6b2dd4c6f1f751ae4b96 to v4-dev.
2018-04-11 21:24:36 +02:00
Alexander Graf
25b8165547 Various minor fixes
* Fix Post.owner_profile in cases where initial owner struct contains id
  only

* Assert node structure is complete enough at Post creation

* Fix get_followees() call in _main()

* Prefer importing from '.' rather than '.<submodule>' in __main__

* Fix name of instaloader_unittests ('-' is illegal in module names)

* Foreport 5fe2a70374b6e81dc875ad33bf44f1001a4f2fc0
  "Fix downloading of hashtags"

* Foreport 67ac8f339716c498cddb2f1489eac6104409646c
  "Enforce being logged in for getting username by id"
2018-04-11 10:49:03 +02:00
André Koch-Kramer
5b9590a768 Story and StoryItem classes to represent stories 2018-04-10 20:29:20 +02:00
Alexander Graf
d90d67d619 Refactor Profile to allow lazy initialization
This changes also allows us to use Profile class in more situations,
which makes the code better.
2018-04-10 15:25:36 +02:00
Alexander Graf
c71be83f43 Split module into multiple files 2018-04-06 17:40:22 +02:00