Timeline for How can I scrape my own content efficiently?
Current License: CC BY-SA 4.0
15 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 8, 2023 at 2:43 | comment | added | C.S.Cameron | So I just need to download the pictures separately. This is Okay as long as Imgur is working. | |
| Jun 7, 2023 at 18:35 | comment | added | Levente | Pictures remaining on Imgur are as safe as "owning" pictures as NFTs :P | |
| Jun 7, 2023 at 13:30 | comment | added | BeastOfCaerbannog |
@C.S.Cameron The HTML and Markdown columns already include the imgur links of your posts' images. So the images are there in essence. You can use the items of the HTML column to save your posts in pure HTML. You can use this simple Python script (that depends on pandas (sudo apt install python3-pandas )) I created to save all posts as HTML files in the folder you run the script from. You can find the script here: pastebin.com/veBEa5Aa Just change the path to your downloaded CSV in the csv_path variable and then run the script using python script.py.
|
|
| Jun 7, 2023 at 12:38 | comment | added | C.S.Cameron | The CSV is nice but I would also like to copy the pictures. like when I do a Save link as > XXXXXXX.htm | |
| Jun 7, 2023 at 12:28 | comment | added | BeastOfCaerbannog |
@C.S.Cameron You can use wget for each link in the CSV to download the entire webpage of each post. I tried to make a script to automate it, but got temporarily blocked for making too many requests (oops!). Perhaps a delay could be used between each wget request to avoid throttling, but I'm not sure what that delay should be and if it would indeed work, so I'm not risking it.
|
|
| Jun 7, 2023 at 10:07 | comment | added | C.S.Cameron | Is there a way to copy and save all selected .htm's at once? , not just links? So they are located on my computer, | |
| Jun 7, 2023 at 5:12 | comment | added | BeastOfCaerbannog | @levente If you adjust the columns as you like with Calc and then save the file as.ods, the columns should remain as you've set them. | |
| Jun 6, 2023 at 21:16 | comment | added | Levente | My only grief is with LibreOffice (6.4): I am interested most in the markdown column, but when LibreOffice opens, the markdown column is the only one that's squished to a 0.3cm width, practically unnoticable, while the cells containing the HTML are so huge (exceeding my screen dimension) that I cannot properly scroll the content: scrolling, in each direction, consistently jumps over specific content. Zooming out helps somewhat. Otherwise I can open the CSV in Gedit, and search for specific words I remember to have used. | |
| Jun 6, 2023 at 20:17 | vote | accept | Levente | ||
| Jun 6, 2023 at 19:34 | history | edited | BeastOfCaerbannog | CC BY-SA 4.0 |
added 1 character in body
|
| Jun 6, 2023 at 19:33 | comment | added | BeastOfCaerbannog | Thanks @RandomPerson ! :) | |
| Jun 6, 2023 at 18:33 | comment | added | Random Person | It's nice to see the detailed steps, like your answers on the main site ;) | |
| Jun 6, 2023 at 18:14 | history | edited | BeastOfCaerbannog | CC BY-SA 4.0 |
added 6 characters in body
|
| Jun 6, 2023 at 17:40 | history | edited | BeastOfCaerbannog | CC BY-SA 4.0 |
deleted 8 characters in body
|
| Jun 6, 2023 at 17:34 | history | answered | BeastOfCaerbannog | CC BY-SA 4.0 |