Commit Graph
100 Commits
Author SHA1 Message Date
Alexander Graf f7e6632f68 Document --tagged{,-only} in docs/cli-options.rst 2018-08-20 10:44:20 +02:00
Alexander Graf a3777cad5d Download profile pic unless --no-profile-pic
Fixes #165.

To download (S = Stories, P = Profile Pic, p = Posts)

p	--no-profile-pic
P	--profile-pic-only
Pp	(default)
S	--no-profile-pic --stories-only
Sp	--no-profile-pic --stories
SP	--stories-only
SPp	--stories
2018-08-13 17:57:28 +02:00
Alexander Graf 998333b7cd Revert "Mention IRC channel #instaloader on Freenode"
This reverts commit 8423e3bd00. The channel did
not have as much activity as expected and an IRC channel does not work if there
are not at least a few people in it.
2018-08-07 19:57:10 +02:00
Alexander Graf dcea0e9672 Release of version 4.0.7 2018-08-05 09:37:18 +02:00
Alexander Graf 809b23a996 doc: Mark Profile.has_higlight_reels as deprecated 2018-08-05 09:13:49 +02:00
Alexander Graf c634f21dcc Fix anon-retry for profiles that blocked viewer
This was broken with 50a5330fec.
2018-08-05 09:01:22 +02:00
Alexander Graf a070ed42ae get_stories(): Fetch in chunks of 100 users
It seems there is a limit of how many user stories can be requested at once.
Now, Instaloader.get_stories() chunks the stories request.

Fixes #157 and fixes #134.
2018-08-05 08:06:53 +02:00
Alexander Graf a34d750152 rate handling: Show %H:%M when request is retried 2018-08-05 08:02:35 +02:00
Alexander Graf 7f81985911 _ArbitraryItemFormatter: Allow chaining attributes
Override get_value() rather than get_field(), to allow chaining of item
attributes, such as --post-metadata-txt={owner_profile.profile_pic_url}.

Motivated by #151.
2018-08-04 11:41:19 +02:00
Alexander Graf 3d4109ded3 travis disable pylint's no-else-return 2018-07-22 11:04:11 +02:00
Alexander Graf 8423e3bd00 Mention IRC channel #instaloader on Freenode 2018-07-22 10:55:29 +02:00
Alexander Graf 5b0f8b75cc Add AUTHORS.md listing Instaloader's authors 2018-07-22 10:54:54 +02:00
Alexander Graf 3cb33603e5 doc: Use Instaloader brand colors for links 2018-07-09 21:32:18 +02:00
Alexander Graf 47db7f7ffa Release of version 4.0.5 2018-07-09 10:18:27 +02:00
Alexander Graf 2567bec975 unittests: Test order of Profile.get_posts()
Motivated by #140.
2018-07-09 10:08:18 +02:00
Alexander Graf 2df1419d68 Fix file extension of profile picture
Fix #142.
2018-07-09 10:07:16 +02:00
Alexander Graf 8211c9daf1 Documentation Improvements
- Fix Spelling
- codesnippets: Improve introduction texts; Write subsection about
  metadata JSON files
2018-06-17 18:10:19 +02:00
Alexander Graf 969d4e3aa4 Post: Write all ever-obtained metadata into JSON
The Post._asdict() code assumed that Post._node always is a subset of
Post._full_metadata_dict. However, it turned out, that `edge_liked_by`,
`thumbnail_src` and `thumbnail_resources` may be contained in Post._node, but
not in Post._full_metadata_dict. Now the keys of both dictionaries are saved in
the JSON file.

Fixes #137.
2018-06-15 17:07:28 +02:00
Alexander Graf 4cf1ae847d Mention donation option in doc
(Suggested by #130)
2018-06-13 08:39:29 +02:00
Alexander Graf 45fe509ac4 doc: Link codesnippets in basic-usage 2018-06-04 12:06:41 +02:00
Alexander Graf ae492ed68b New doc section: codesnippets / Advanced Examples
Presents code examples that use the instaloader module for more advanced tasks
than what is possible with the Instaloader command line interface.

Presents #46, #56, #110, #113, #120, #121.
2018-06-04 11:57:28 +02:00
Alexander Graf 0e433294ad unittests: Preserve query timestamps between tests
This avoids getting rate limited while running all the tests.
2018-05-30 13:16:16 +02:00
Alexander Graf a807408d8f Release of version 4.0.3 2018-05-23 08:57:36 +02:00
Alexander Graf 86035b8d2d Fix obtaining csrftoken for login
Fixes #122. Further fixes processing of login JSON response regarding error
handling.
2018-05-23 08:50:37 +02:00
Alexander Graf a3eb93c21a Profile: Cache iphone data structure if queried
Profile.get_profile_pic_url() and Profile.has_highlight_reels did the same
query without remembering the response, resulting in that query being made
twice when downloading stories and profile pic of a profile.
2018-05-19 09:52:07 +02:00
Alexander Graf 60d35a95df docs: note that --stories-only @login is bad
Motivated by #112, #118.
2018-05-14 11:04:48 +02:00
Alexander Graf 920dcd0407 Have Profile.{followers,followees,external_url} 2018-05-14 10:46:04 +02:00
Alexander Graf d19dd2c9d4 Fix internal references in docs/as-module
Further, document all attributes that are referred to, and ensure that there
will be no broken references again by letting Travis call Sphinx with -W -n.
2018-05-14 10:46:04 +02:00
Alexander Graf be43ebb5c5 Unittest for Profile.has_viewable_story 2018-05-14 10:42:31 +02:00
Alexander Graf 52eb6d8e27 Fix minor issues regarding last commit
- Close session from InstaloaderContext.anonymous_copy().

- Have graphql_count_per_slidingwindow default value in one place rather than
  three.
2018-05-14 10:27:34 +02:00
Alexander Graf b8797cf337 Release of version 4.0.1 2018-05-08 19:16:55 +02:00
Alexander Graf fa06e90e09 Fix anonymous retry logic 2018-05-08 19:12:42 +02:00
Alexander Graf d464396d6e Adjust rate control behavior
Increment graphql_count_per_slidingwindow to 25; Let context._sleep() choose
sleep times from upper-bounded exponential distribution with similar expected
value as before, letting wait times between queries seem more natural.
2018-05-08 18:54:28 +02:00
Alexander Graf 9961312d34 Fix anonymous retry on profile that blocked viewer 2018-05-08 18:15:17 +02:00
Alexander Graf 211b2269ff download_profile: Request story only if has_highlight_reel
This is a simple modification to save GraphQL queries, hence to more likely
stay within rate limits without having to wait between profiles.

Fixes #112.
2018-05-08 18:10:28 +02:00
Alexander Graf b3df5a5f07 Fix bugs in Instaloader.update_comments()
- Cast id and created_at from JSON to int to be compatible with comments JSON
  files from Instaloader 3.3,
- Do not fail if comment JSON files are empty,
- Close file descriptors of comment JSON files when reading,
- Do no create empty comment JSON files if generating content fails.

Closes #114.
2018-05-08 17:39:00 +02:00
Alexander Graf 5204990db9 Release of version 4.0 2018-05-02 17:38:07 +02:00
Alexander Graf 94df221da2 Merge branch 'v4-dev' 2018-05-02 17:29:12 +02:00
Alexander Graf e7decba5ee main: Enumerate profiles separated with ' ' in log 2018-05-02 17:27:46 +02:00
Alexander Graf a3a09e1952 Very minor tweak to a paragraph in basic-usage.rst 2018-05-02 17:26:53 +02:00
Alexander Graf b1e55a41a4 Do not log Post's username in download_saved_posts
This improves speed, as get_saved_posts() returns Posts that do not have
owner_profile info. So, printing the owner_username often required an
additional request.
2018-05-01 20:22:52 +02:00
Alexander Graf b5ad92e236 Trivial improvement to Profile.from_id() 2018-05-01 19:52:39 +02:00
Alexander Graf 084cfb102c Handle ValueError on bad profile id file
Fixes "ValueError: invalid literal for int() with base 10: b''", reported in #95.
2018-05-01 15:14:19 +02:00
Alexander Graf 2722da6ae4 Fix anon username from id (find renamed profile)
maybe related to #95, #104, 67ac8f3397.
2018-05-01 13:11:40 +02:00
Alexander Graf 1469064176 Minor documentation tweaks
- Instaloader as Cronjob sections

- Many new internal links to ease navigation
2018-04-30 09:56:43 +02:00
Alexander Graf 18ef484f3a First Release Candidate for Version 4.0 2018-04-29 22:53:20 +02:00
Alexander Graf dba74ab8d4 Minor documentation improvements 2018-04-29 16:41:24 +02:00
Alexander Graf dd80ef484c Improve formatting of basic-usage/what-to-download 2018-04-29 16:26:31 +02:00
Alexander Graf 6117e223a7 Describe Metadata Text Files Feature 2018-04-29 16:24:55 +02:00
Alexander Graf 7c76202f1c Update documentation regarding v4 changes
as-module.rst: List how Post and Profile is created

Describe new filename specification logic

Update filter-posts section
2018-04-29 13:33:25 +02:00
Alexander Graf 5036cc7b44 main: Ignore profile JSON files as target
It might annoy or be undesired when using Instaloader with a globbed
JSON file list, i.e. instaloader profile/*.json to just update all post
metadata.
2018-04-29 11:22:56 +02:00
Alexander Graf 8a8ea2913c Improve post/storyitem-metadata-txt behavior
If an expression evaluates to None, an empty string is yielded, rather
than 'None'.

Blanks (and newlines) are strip()ed from the formatted strings, which
also avoids creation of whitespace-only textfiles.

save_captions parameter to Instaloader constructor has been removed, as
it is equivalent to setting both {post,storyitem}_metadata_txt_pattern
to '' (empty string).

Instaloader.save_caption() now prints the '[{old}] updated [{new}]' if
the caption was updated, rather than printing the new caption twice,
which was confusing.
2018-04-29 11:17:48 +02:00
Alexander Graf b2dd395a86 Link cli-options in Instaloader __init__ docstring 2018-04-29 10:13:31 +02:00
Alexander Graf d21fb4154e fine-tune as-module.rst 2018-04-28 21:45:57 +02:00
Alexander Graf 7edc69454d doc: Fix links to Instaloader classes 2018-04-28 19:50:14 +02:00
Alexander Graf 147cd1a580 Post_get_comments() yield namedtuple PostComment 2018-04-28 18:17:00 +02:00
Alexander Graf 0e40da1c70 Let Post.get_likes() yield Profile instances 2018-04-28 17:32:13 +02:00
Alexander Graf 4dcb23cfe5 {Post,Profile,StoryItem}.get_node() -> ._as_dict()
This is consistend with namedtuple._as_dict() and marks it as a
protected member, which is good as it should be only accessed by
save_structure_to_file().
2018-04-28 17:15:42 +02:00
Alexander Graf 3dcfcff04c namedtuples PostLocation and PostSidecarNode
Post.get_sidecar_edges() has been renamed to Post.get_sidecar_nodes().

Post.get_location() has been made to a property Post.location and it now
internally stores location struct and makes it JSON store-/loadable.
2018-04-28 17:11:08 +02:00
Alexander Graf 2c50972e08 Update as-module.rst and docstrings 2018-04-28 15:47:04 +02:00
Alexander Graf b9ac40899d Theme-related Documentation Enhancements
- Consistent background color in all types of codes
- Improve dd (list item) padding
- Link to Index in global TOC in navbar
- Briefly describe Instaloader in `index.html` title
- Remove page contents (navbartoc) in index, genindex and installation
- Link to new pypi.org rather than old pypi.python.org
2018-04-28 15:47:04 +02:00
Alexander Graf 57ce51d4ef Support datetime objects in filter strings
Now --only-if="date_utc<datetime(2018,1,1,hour=12)" is possible.
2018-04-19 15:42:26 +02:00
Alexander Graf 7d2830b8b2 Have --storyitem-filter (--only-if for StoryItems) 2018-04-19 15:12:05 +02:00
Alexander Graf 7688bdce45 Reverse story download order to enable fast-update
Port of 729aa31d86 to v4-dev branch.
2018-04-18 16:38:13 +02:00
Alexander Graf eecaca7c7b Release of version 3.3.9 2018-04-18 16:25:53 +02:00
Alexander Graf 729aa31d86 Reverse story download order to enable fast-update
Fixes #100.
2018-04-18 16:24:47 +02:00
Alexander Graf 94df60292e Fix bug about 429 handling
It caused Instaloader to fail with a TypeError on 429 Too Many Requests.
2018-04-18 16:24:20 +02:00
Alexander Graf 2d9acd9989 Slightly improve error handling
Use raise from rather than raise where appropriate

Have a logic to reduce graphql page length if 400 Bad Request is
returned
2018-04-18 15:58:45 +02:00
Alexander GrafandGitHub 09f2eb3601 Set GRAPHQL_PAGE_LENGTH to 12
This is the length the browser requests. Might alleviate #94.
2018-04-18 11:41:00 +02:00
Alexander Graf 066c3de113 get_followe{e,r}s yield Profiles rather than dicts 2018-04-17 15:11:50 +02:00
Alexander Graf 9d8175354b Add paging tests for profiles and stories 2018-04-17 14:23:12 +02:00
Alexander Graf dcd16f29c2 Split tests TestInstaloader{Anonymously,LoggedIn} 2018-04-17 14:10:27 +02:00
Alexander Graf 5d513da73c Fix x-instagram-gis calculation
Port of f16b99bec0 to v4-dev branch.
2018-04-17 14:01:11 +02:00
Alexander Graf 3e904e8a14 Release of version 3.3.8 2018-04-17 13:30:45 +02:00
Alexander Graf f16b99bec0 Fix GraphQL Queries (again)
Fixes #96.
2018-04-17 13:17:50 +02:00
Alexander Graf 59152214cf Allow specification of what to write in txt files 2018-04-13 22:54:40 +02:00
Alexander Graf 80f701150c New formatter for filename pattern 2018-04-13 21:19:04 +02:00
Alexander Graf df1cdb5d48 Remove now-unneeded Tristate 2018-04-13 19:07:23 +02:00
Alexander Graf 7976a4811e Fix x-instagram-gis calulation
Port of efa0b93fda to v4-dev branch.
2018-04-13 18:34:35 +02:00
Alexander Graf d316ddf02a Release of version 3.3.7 2018-04-13 18:17:40 +02:00
Alexander Graf efa0b93fda Fix rhx_gis calculation to fix 403 (fix #94) 2018-04-13 18:05:36 +02:00
Alexander Graf 8ab7d3398f Rename Instaloader.download_story{,item}
It makes more sense since the object is called StoryItem.
2018-04-12 22:04:06 +02:00
Alexander Graf f0bebd0d96 Saving and reimporting of JSON files
Metadata JSON files are now created by default. They can later be given
as a target to redownload Posts, StoryItems or Profiles with new
settings.
2018-04-12 22:03:02 +02:00
Alexander Graf 5d249c5401 Fix bug concerning Post class 2018-04-12 17:18:53 +02:00
Alexander Graf 9b3014d5bf Fix anonymous GraphQL queries
Port of 73ec884ea4 to v4-dev.
2018-04-11 21:24:36 +02:00
Alexander Graf 020830d591 Allow graphql_node_list() to take a first page 2018-04-11 20:44:17 +02:00
Alexander Graf 74d5e35eb8 get_{comments,likes}: save requests
Post.get_{comments,likes} had a logic to save requests if comments/like
count is low. That logic had a bug that is now fixed.

Related to #94.
2018-04-11 11:24:49 +02:00
Alexander Graf 25b8165547 Various minor fixes
* Fix Post.owner_profile in cases where initial owner struct contains id
  only

* Assert node structure is complete enough at Post creation

* Fix get_followees() call in _main()

* Prefer importing from '.' rather than '.<submodule>' in __main__

* Fix name of instaloader_unittests ('-' is illegal in module names)

* Foreport 5fe2a70374
  "Fix downloading of hashtags"

* Foreport 67ac8f3397
  "Enforce being logged in for getting username by id"
2018-04-11 10:49:03 +02:00
Alexander Graf 8612712f44 Release of version 3.3.5 2018-04-10 20:57:53 +02:00
Alexander Graf 67ac8f3397 Enforce being logged in for getting username by id
It seems it is not possible at the moment to get username by userid
anonymously.
2018-04-10 20:52:39 +02:00
Alexander Graf 5fe2a70374 Fix downloading of hashtags 2018-04-10 20:34:07 +02:00
Alexander Graf d90d67d619 Refactor Profile to allow lazy initialization
This changes also allows us to use Profile class in more situations,
which makes the code better.
2018-04-10 15:25:36 +02:00
Alexander Graf 8bd9c4449c Theme-related Documentation Enhancements
- Consistent background color in all types of codes
- Improve dd (list item) padding
- Link to Index in global TOC in navbar
- Briefly describe Instaloader in `index.html` title
- Remove page contents (navbartoc) in index, genindex and installation
- Link to new pypi.org rather than old pypi.python.org
2018-04-09 16:07:08 +02:00
Alexander Graf c7f6cff15a basename(argv[0]) instead of argv[0] in usage str 2018-04-06 17:40:33 +02:00
Alexander Graf c71be83f43 Split module into multiple files 2018-04-06 17:40:22 +02:00
Alexander Graf 3511811090 Move Instaloader.main method to module-level _main 2018-04-06 17:40:22 +02:00
Alexander Graf c350847b50 InstaloaderContext class for log and low-level IO 2018-04-06 17:40:13 +02:00
Alexander Graf 8623829696 Strip trailing '/' chars from target specifiers
This makes Instaloader more shell-autocompletion-friendly.
2018-03-31 15:18:05 +02:00
Alexander Graf 235522dfd7 Merge branch 'master' into v4-dev 2018-03-31 15:06:39 +02:00
Alexander Graf 5b6e8f1a31 Fix indent in USAGE_STRING message 2018-03-29 11:19:15 +02:00