warc crawling - SERP Analysis

https://commoncrawl.org/latest-crawl

https://commoncrawl.org/blog/navigating-the-warc-file-format

https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.0/

https://github.com/ArchiveTeam/grab-site

https://community.preservica.com/how-to-guides-70/getting-started-with-preservica-starter-how-to-ingest-a-warc-file-159

https://www.marginalia.nu/log/94_warc_warc/

https://news.ycombinator.com/item?id=38729684

https://github.com/helgeho/Web2Warc

https://crawler.docs.browsertrix.com/user-guide/common-options/

https://stackoverflow.com/questions/46307663/common-crawl-getting-warc-file

https://news.ycombinator.com/item?id=31799147

https://code402.com/blog/hello-warc-common-crawl-code-samples/

https://ad-publications.cs.uni-freiburg.de/student-projects/easy-huc/common_crawl.html

https://cwiki.apache.org/confluence/display/nutch/CommonCrawlDataDumper

https://blog.christianperone.com/2023/06/appreciating-llms-data-pipelines/

https://groups.google.com/g/common-crawl/c/7MuqVmvajoA

https://specialcollections.blogs.bristol.ac.uk/2022/03/11/archiving-the-web-collecting-web-crawls-for-the-university-archive/

https://crawler.docs.browsertrix.com/user-guide/outputs/

https://forum.webrecorder.net/t/i-have-a-problem-with-the-page-index-in-a-warc-file/481

https://support.archive-it.org/hc/en-us/articles/360000762966-WARC-Naming-Conventions

https://dmorgan.info/posts/common-crawl-python/

https://openpreservation.org/blogs/crawling-offline-web-content-the-nl-menu-case/

https://groups.google.com/g/common-crawl/c/HY2OIyg-RGA

https://forum.webrecorder.net/t/where-in-browsertrix-crawler-code-is-the-warc-stored/386

https://netpreserve.org/resources/WAC25_POSTER18_NAGEL.pdf

https://scholarworks.sjsu.edu/etd_projects/1039/

https://freeman.vc/notes/parsing-common-crawl-in-a-day-for-60

https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/download.html

https://lil.law.harvard.edu/blog/2024/04/04/a-modest-improvement-to-warc-gpt/

https://medium.com/snowflake/processing-webarchive-warc-files-natively-in-snowflake-fd973217dd96

https://support.archive-it.org/hc/en-us/articles/360015225051-How-to-find-and-download-your-WARC-files-with-WASAPI

https://www.tsl.texas.gov/slrm/blog/2021/10/archives-month-web-archive-warc-file-format/

https://anjackson.net/zombse/032013%20Digital%20Preservation/static/questions/101.html

https://datos.gob.es/en/blog/open-data-world-common-crawl-frees-petabytes-web-data

https://guides.lib.vt.edu/c.php?g=749461&p=6903942

https://www.warc.com/

https://n0tan3rd.github.io/Squidwarc/

https://skeptric.com/text-meta-data-commoncrawl/

https://docs.marginalia.nu/2_crawling/5_warc_export/

https://www.kaggle.com/code/harshsinghal/download-sample-warc-files-common-crawl

https://repo1.maven.org/maven2/com/digitalpebble/stormcrawler/storm-crawler-warc/1.5.1/

https://pywb.readthedocs.io/en/latest/manual/indexing.html

https://digitalcorpora.org/corpora/file-corpora/cc-main-2021-31-pdf-untruncated/constructing-the-cc-main-2021-31-pdf-untruncated-corpus/

https://www.loc.gov/preservation/digital/formats/fdd/fdd000236.shtml

https://registry.opendata.aws/commoncrawl/

https://avilpage.com/2022/11/common-crawl-laptop-extract-subset.html

https://www.dpconline.org/docs/miscellaneous/events/2020-events/2394-2020-webarch-s3-workflows

https://www.reddit.com/r/Archiveteam/comments/uz7afo/help_with_warc_filesextracting_a_portion_of_a/

https://hackage.haskell.org/package/warc-1.0.1/docs/Data-Warc.html

https://nlpl.eu/skeikampen23/nagel.230206.pdf

https://cran.r-project.org/package=sparkwarc

https://git.jordan.im/crawl/tree/README.md

https://datascience.fm/extracting-the-top-used-js-libraries-from-common-crawl/

https://anjackson.net/blog/uwa/8.19.2015,%208.47.54%20PM%20Twine%20Archive.html

https://medium.com/%40pradeepgoel/common-crawl-open-data-set-tool-99b10369ed42

https://netpreserve.org/ga2019/wp-content/uploads/2019/07/IIPCWAC2019-SEBASTIAN_NAGEL-Accessing_WARC_files_via_SQL-poster.pdf

https://pkg.go.dev/git.jordan.im/crawl/warc

https://lil.law.harvard.edu/blog/2024/02/12/warc-gpt-an-open-source-tool-for-exploring-web-archives-with-ai/

https://scholarworks.sjsu.edu/cgi/viewcontent.cgi?article=2039&context=etd_projects

https://gitee.com/bit212/crawl-warc-go?skip_mobile=true

https://mvnrepository.com/artifact/com.digitalpebble.stormcrawler/storm-crawler-warc

https://aws.amazon.com/marketplace/pp/prodview-zxtb4t54iqjmy

https://skeptric.com/notebooks/WAT%20WET%20WARC%20-%20Common%20Crawl%20Archives.html

http://terrier.org/docs/v4.2/javadoc/org/terrier/indexing/WARC09Collection.html

https://twitter.com/jeffbarr/status/970746271764893698

https://arxiv.org/html/2411.05409v1

https://www.duo.uio.no/bitstream/handle/10852/57836/Kristoffersen_MSc2.pdf

https://repositum.tuwien.at/handle/20.500.12708/52990

https://www.jdsupra.com/legalnews/warc-and-worm-digital-storage-web-53918/

https://anarc.at/services/archive/web/

https://www.reddit.com/r/mlscaling/comments/1c9xcde/fineweb_15t_tokens_of_cleaned_common_crawl/

https://dzone.com/articles/need-billions-of-web-pages-dont-bother-crawling

https://bugs.debian.org/915372

https://tech.marksblogg.com/petabytes-of-website-data-spark-emr.html

https://core.ac.uk/outputs/484276257/?source=1&algorithmId=15&similarToDoc=&similarToDocKey=URL&recSetID=7ac42fd9-10c0-4c73-b77d-03ff7cff930c&position=5&recommendation_type=same_repo&otherRecs=643425457%2C249325321%2C229273961%2C127867367%2C484276257

https://www.vancouverarchives.ca/2017/05/18/preserving-the-citys-website-with-archive-it/

https://www.dpconline.org/blog/bit-list-blog/the-long-warc-to-freedom

http://events17.linuxfoundation.org/sites/events/files/slides/aceu2016-snagel-crawling-web-commoncrawl.pdf

https://www.forbes.com/sites/kalevleetaru/2017/09/28/common-crawl-and-unlocking-web-archives-for-research/

https://www.netwerkdigitaalerfgoed.nl/wp-content/uploads/2020/06/NDE-Webinar-Webarchivering-18-juni-2020-Jesse-de-Vos-Crash-course-webarchivering.pdf

https://huggingface.co/datasets/Symato/cc/blob/d034e0fc09b6feb31ed61fb186bfe5b3571f49f5/crawl-data/CC-MAIN-2017-26/segments/1498128320476.39/warc/CC-MAIN-20170625083108-20170625103108-00239.jsonl.xz

https://digitalcommons.odu.edu/cgi/viewcontent.cgi?article=1154&context=computerscience_fac_pubs

https://blog.entropic-data.com/2018/05/08/defining-data-science-using-the-common-crawl-web-corpus-1/

http://crawler.archive.org/articles/releasenotes/1_10_0.html

http://digitalpebble.blogspot.com/2017/03/need-billions-of-web-pages-dont-bother_29.html

https://heritrix.readthedocs.io/en/latest/glossary.html

https://indico.cern.ch/event/883268/contributions/3995764/attachments/2204558/3729718/2020-10-13--1150--Presentation--Nagel-S--osssym2020-sn-web-archives.pdf

http://stackoverflow.com/a/27178219

https://www.jcchouinard.com/python-commoncrawl-extraction/

https://git.jordan.im/crawl/

https://repo1.maven.org/maven2/com/digitalpebble/stormcrawler/storm-crawler-warc/

https://m.youtube.com/watch?v=MQUMIWbJLo4

https://www.trackawesomelist.com/iipc/awesome-web-archiving/readme/

https://huggingface.co/datasets/Symato/cc/blob/d1892052e8fa689ea6977dbc09aabfe688f3f035/crawl-data/CC-MAIN-2018-39/segments/1537267156857.16/warc/CC-MAIN-20180921053049-20180921073449-00113.jsonl.xz

https://cran.r-project.org/web/packages/sparkwarc/sparkwarc.pdf

https://superuser.com/questions/14403/how-can-i-download-an-entire-website

http://www.tofuproxy.stargrave.org/WARCs.html

https://en.wikipedia.org/wiki/WARC_(file_format)

https://alternativeto.net/software/grab-site/about/