DataExcavator - quick web scraper
https://data-excavator.com
Readme - Operating Manual
Doc version 1.0 from 19 Apr 2020

## 1. System requirements

- OS: Microsoft Windows 7+ and above
- Installed .NET Framework 4.5.2 or 4.6
- ~300 Mb space on HDD
- ~300 Mb RAM and above
- Stable internet connection
- Firewall: opened port 80, 443
- CPU with 2 cores and above
- More than 1Gb free RAM memory
- Some free space on HDD
- MS Visual Studio 2017 and above

## 2. Programming language and software components

- Written in C# (.NET)
- Based on CEF (Chromium Embedded Framework)
- Uses next libraries: EPPlus, HtmlAgilityPack, ExcavatorSharp, CEFSharp, RestSharp, Newtonsoft.JSON, log4net

## 3. Use cases

We assume that you may have downloaded this package for two reasons.
Case #1: You are an ordinary user who wants to use our program for data scraping. In this case, you do not need to study this manual, but just download the installation file and run it. This manual is intended only for developers.
The link to download the installation package can be found either on our website https://data-excavator.com
Your license key is: Vbk4eQWp8kdmqnl2QlzWkBWIQzu++xD6yEwYB68SiEFVSOyRL0fEB0T7XlheB93/rRWdFtsnoHeiUu0WcVYHqqCZzPq0s0APf6KkND8B3N6ZL0yZ+vQsvvTCFf+SYADDMcW1RQLKHh+r03w+BOulu6nCM0sSHGDtqSiUGpjSa1RA4DLdSnGW7pTbXuqI2CGs
Case #2: You are a developer who wants to do something based on this code to solve his tasks. In this case, read the instruction carefully.

## 4. Developer start guide

1. Download and install MS Visual Studio 2017+ and .NET Framework 4.5.2 or 4.6
2. Launch Visual Studio
3. Open "Source/DataExcavator.sln" project
4. Click on "Buid" -> "Rebuild solution" and wait until Visual Studio makes rebuild
5. Check is any errors displayed into Errors list
6. If you see any errors in Errors list, just reopen the solution

The entry point to the application is the MainWindow.xaml.cs file. This is where the interface part and kernel logic are initialized. An equally important module worth mentioning is the UIControls/DEProjectCube.cs file - it contains the interface of the scraping task that can be created in the program.

## 5. Using core library directly

DataExcavator is based on ExcavatorSharp program library. If you want to use it directly, without our interface, you can do so. Below we will give you examples of working with the library directly. You can always download the library via nuget, or get its current version on our website.
In general, our interface directly implements all functions of the native library, and it will probably be convenient for you to study the work of our interface first, and then deal with direct calls to the library.
Additional documentation also available here: https://data-excavator.com/excavatorsharp-docs/
Please note that nuget x64 build is intended to run strictly on x64 platforms. If you are looking for an x86 build, please use our website to directly download the x86 library. The build is not cross-platform (AnyCPU) based on the peculiarities of Chromium Embedded Framework.

## 5.1. ExcavatorSharp architecture

## 5.2. ExcavatorSharp elementary example

The following example shows how our library can parse walmart.com. Here we simply bypass all product pages and get data for each product - name and price.

    public class ESHelloWorld
    {
        public void ParseWalmartAnyPage()
        {
            //1. Initialize tasks storage
            CEFSharpFactory.InitializeCEFBrowser();
            DataExcavatorTasksFactory TasksFactory = new DataExcavatorTasksFactory();
            TasksFactory.InitializeExcavator("Vbk4eQWp8kdmqnl2QlzWkBWIQzu++xD6yEwYB68SiEFVSOyRL0fEB0T7XlheB93/rRWdFtsnoHeiUu0WcVYHqqCZzPq0s0APf6KkND8B3N6ZL0yZ+vQsvvTCFf+SYADDMcW1RQLKHh+r03w+BOulu6nCM0sSHGDtqSiUGpjSa1RA4DLdSnGW7pTbXuqI2CGs");

            //2. Create your first task
            DataExcavatorTask NewTask = new DataExcavatorTask(
                "My first scraping task",
                "https://www.walmart.com/",
                "Scrape some data from walmart",
                new List<DataGrabbingPattern>() {
                    new DataGrabbingPattern() {
                        PatternName = "Apple ipads pattern",
                        AllowedPageUrlsSubstrings = new string[] { "ip/" },
                        GrabbingItemsPatterns= new List<DataGrabbingPatternItem>() {
                            new DataGrabbingPatternItem("Item name", new GrabberSelector("h1.prod-ProductTitle", DataGrabbingSelectorType.CSS_Selector)),
                            new DataGrabbingPatternItem("Item price", new GrabberSelector(".prod-PriceSection .display-inline-block-m .price-characteristic", DataGrabbingSelectorType.CSS_Selector))
                        }
                    }
                },
                new CrawlingServerProperties() { RespectRobotsTxtFile = false, RespectOnlySpecifiedUrls = new string[] { "ip/" }, PrimaryDataCrawlingWay = DataCrawlingType.NativeCrawling },
                new GrabbingServerProperties(),
                "C:/ExcavatorSharp/FirstProject"
            );

            //3. Subscribe events
            NewTask.PageCrawled += NewTask_PageCrawled;
            NewTask.PageGrabbed += NewTask_PageGrabbed;

            //4. Add links to crawling and start task
            TasksFactory.AddTask(NewTask);
            NewTask.AddLinksToCrawling(new List<string>() { "https://www.walmart.com/ip/Apple-10-2-inch-iPad-7th-Gen-Wi-Fi-32GB/216119597" });
            NewTask.StartTask();
        }

        /// <summary>Fires when some data from website been extracted</summary>
        private void NewTask_PageGrabbed(PageGrabbedCallback GrabbedPageData)
        {
            Dictionary<DataGrabbingPattern, DataGrabbingResult> GrabbedResults = GrabbedPageData.GrabbingResults;
            //Do what you want with grabbed data
        }

        /// <summary>Fires when some page from website been downloaded and parsed to structured HTML</summary>
        private void NewTask_PageCrawled(PageCrawledCallback CrawledPageData)
        {
            //... just take crawled data or skip this event. Additionaly, you can prevent page grabbing if you want (CrawledPageData.PreventPageGrabbing = true;)
        }
    }

## 5.3. Data export

As we have already written, by default the program does not require you to use an event model or deep integration, nor does it require a deep understanding of the algorithms. By default, the library will simply run with a set of parameters, after which it will start retrieving data and saving it to your hard drive. Next, you can export this data in a convenient format at any time.
The following example shows how you can export data and do without an event model.

    /// <summary>
    /// Data export testing
    /// </summary>
    public class ESExportExample
    {
        public void ParseAndExportWalmartAnyPage()
        {
            //1. Initialize tasks storage
            CEFSharpFactory.InitializeCEFBrowser();
            DataExcavatorTasksFactory TasksFactory = new DataExcavatorTasksFactory();
            TasksFactory.InitializeExcavator("Vbk4eQWp8kdmqnl2QlzWkBWIQzu++xD6yEwYB68SiEFVSOyRL0fEB0T7XlheB93/rRWdFtsnoHeiUu0WcVYHqqCZzPq0s0APf6KkND8B3N6ZL0yZ+vQsvvTCFf+SYADDMcW1RQLKHh+r03w+BOulu6nCM0sSHGDtqSiUGpjSa1RA4DLdSnGW7pTbXuqI2CGs");

            //2. Create your first task
            DataExcavatorTask NewTask = new DataExcavatorTask(
                "My first scraping task",
                "https://www.walmart.com/",
                "Scrape some data from walmart",
                new List<DataGrabbingPattern>() {
                    new DataGrabbingPattern() {
                        PatternName = "Apple ipads pattern",
                        AllowedPageUrlsSubstrings = new string[] { "ip/" },
                        GrabbingItemsPatterns= new List<DataGrabbingPatternItem>() {
                            new DataGrabbingPatternItem("Item name", new GrabberSelector("h1.prod-ProductTitle", DataGrabbingSelectorType.CSS_Selector)),
                            new DataGrabbingPatternItem("Item price", new GrabberSelector(".prod-PriceSection .display-inline-block-m .price-characteristic", DataGrabbingSelectorType.CSS_Selector))
                        }
                    }
                },
                new CrawlingServerProperties() { RespectRobotsTxtFile = false, RespectOnlySpecifiedUrls = new string[] { "ip/" }, PrimaryDataCrawlingWay = DataCrawlingType.NativeCrawling },
                new GrabbingServerProperties(),
                "C:/ExcavatorSharp/FirstProject"
            );

            //3. Add links to crawling and start task
            TasksFactory.AddTask(NewTask);
            NewTask.AddLinksToCrawling(new List<string>() { "https://www.walmart.com/ip/Apple-10-2-inch-iPad-7th-Gen-Wi-Fi-32GB/216119597" });
            NewTask.StartTask();

            //... Wait some time - 30 seconds, for example ...

            Thread.Sleep(30000);

            //Exporting by dates period
            NewTask.ExportAllGrabbedData(
                "C:/ExportPath",
                DataExportingFormat.XLSX,
                DataExportingType.OuterHTML,
                ",",
                new DateTime(2019, 10, 10),
                DateTime.Now
            );

            //Exporting by selected meta data entries. Files with metadata will be generated by GrabbingServer
            List<GrabbedPageMetaInformationDataEntry> AllParsedDataSavedOnHDD = NewTask.GetGrabbedDataListOverview();
            NewTask.ExportSelectedGrabbedData(
                "C:/ExportPath2",
                DataExportingFormat.JSON,
                DataExportingType.InnerText,
                ",",
                AllParsedDataSavedOnHDD
            );
        }
    }

## 5.4. Scraping images and binary content

    /// <summary>
    /// Extract data from img tags
    /// </summary>
    class ImagesScraping
    {
        public void ScrapeImages()
        {
            //1. Initialize tasks storage
            CEFSharpFactory.InitializeCEFBrowser();
            DataExcavatorTasksFactory TasksFactory = new DataExcavatorTasksFactory();
            TasksFactory.InitializeExcavator("Vbk4eQWp8kdmqnl2QlzWkBWIQzu++xD6yEwYB68SiEFVSOyRL0fEB0T7XlheB93/rRWdFtsnoHeiUu0WcVYHqqCZzPq0s0APf6KkND8B3N6ZL0yZ+vQsvvTCFf+SYADDMcW1RQLKHh+r03w+BOulu6nCM0sSHGDtqSiUGpjSa1RA4DLdSnGW7pTbXuqI2CGs");

            //2. Create some pattern
            DataGrabbingPattern NewPattern = new DataGrabbingPattern();
            NewPattern.AllowedPageUrlsSubstrings = new string[] { "page1" };
            NewPattern.PatternName = "Grab images from item page";
            NewPattern.GrabbingItemsPatterns = new List<DataGrabbingPatternItem>();

            //3. Target to image attribute - download data from SRC tag and save to file
            DataGrabbingPatternItem PatternItem = new DataGrabbingPatternItem(
                    "Image tag",
                    new GrabberSelector("img.item-image", DataGrabbingSelectorType.CSS_Selector),
                    true,
                    new ParsingBinaryAttributePattern[] {
                        new ParsingBinaryAttributePattern("src", true, true)
                    }
                );
        }
    }

## 5.5. Interacting with websites using JS

Using our library, you can interact with sites through JavaScript. Please note that the interaction is not linear (such as in Selenium). You cannot consistently perform an endless set of actions on a page. Our library, however, is closer to the search engines and its main task is to quickly scan and parse data from the site pages. If you are facing the task of narrowly parsing a small number of pages, for example – “Go to the site, enter your login and password, go to the first page, set a filter, go to another page, set another filter, click on the deitpicker, then go to the third page, click on the download” – then you will not be very convenient to use our solution. In other cases our library will be quite applicable, convenient and effective.
For interaction with pages of a site the set CEFBehaviors is used. Each element of this set defines the interaction of our library with the target site.

    /// <summary>
    /// Makes behaviors testing
    /// </summary>
    public void TestCEFBehaviors()
    {
        //1. Initialize tasks storage
        CEFSharpFactory.InitializeCEFBrowser();
        DataExcavatorTasksFactory TasksFactory = new DataExcavatorTasksFactory();
        TasksFactory.InitializeExcavator("Vbk4eQWp8kdmqnl2QlzWkBWIQzu++xD6yEwYB68SiEFVSOyRL0fEB0T7XlheB93/rRWdFtsnoHeiUu0WcVYHqqCZzPq0s0APf6KkND8B3N6ZL0yZ+vQsvvTCFf+SYADDMcW1RQLKHh+r03w+BOulu6nCM0sSHGDtqSiUGpjSa1RA4DLdSnGW7pTbXuqI2CGs");

        //2. Define primary crawling way
        CrawlingServerProperties CrawlerProps = new CrawlingServerProperties();

        //Use Chromium Embedded Framework to download pages (dynamic content SUPPORTED, JS-interfaction SUPPORTED, more slowly crawling)
        CrawlerProps.PrimaryDataCrawlingWay = DataCrawlingType.CEFCrawling;

        //Define a set of CEF behaviors
        CrawlerProps.CEFCrawlingBehaviors = new List<CEFCrawlingBehavior>();
        CrawlerProps.CEFCrawlingBehaviors.Add(new CEFCrawlingBehavior(
                "/item/",
                10,
                "$(function() { $('show-more-data').click(); });",
                20,
                CEFCrawlingPageLeaveEventType.LeavePageAfterJSEventReturnsSomeResult,
                "function a() { if ($('page-new-links').length == 0) return 'PAGE_INDEXED'; } a();",
                "PAGE_INDEXED"
            ));

        /*
            1. Navigate page with some URL contains /item/
            2. Wait 10 seconds
            3. Execute on page "$(function() { $('show-more-data').click(); });"
            4. Wait 20 seconds
            5. GET PAGE ACTUAL RENDERED HTML CODE
            6. Execute on page "function a() { if ($('page-new-links').length == 0) return 'PAGE_INDEXED'; } a();"
            7. Check results of a() function. If results is PAGE_INDEXED -> stop page crawling. Otherwise, go to step 2
            */

        //3. Create task
        DataExcavatorTask NewTask = new DataExcavatorTask(
            "Task1",
            "https://www.walmart.com",
            "Some description",
            new List<DataGrabbingPattern>(),
            CrawlerProps,
            new GrabbingServerProperties(),
            "C:/Walmart1"
        );
    }

## 5.6. Proxies support

Our library supports data proxying. You can define the proxy pool to be used for data scanning. In this case, each time you finish downloading the current page, the crawling server will use a specific proxy from this pool. The choice of proxy will depend on the rotation algorithm. You can use a sequential rotation algorithm, or a random rotation (random) algorithm.

    /// <summary>
    /// Proxies testing
    /// </summary>
    class ProxiesSupport
    {
        public void TestProxy()
        {
            //1. Initialize tasks storage
            CEFSharpFactory.InitializeCEFBrowser();
            DataExcavatorTasksFactory TasksFactory = new DataExcavatorTasksFactory();
            TasksFactory.InitializeExcavator("Vbk4eQWp8kdmqnl2QlzWkBWIQzu++xD6yEwYB68SiEFVSOyRL0fEB0T7XlheB93/rRWdFtsnoHeiUu0WcVYHqqCZzPq0s0APf6KkND8B3N6ZL0yZ+vQsvvTCFf+SYADDMcW1RQLKHh+r03w+BOulu6nCM0sSHGDtqSiUGpjSa1RA4DLdSnGW7pTbXuqI2CGs");

            //2. Define proxy list
            CrawlingServerProperties CrawlerProps = new CrawlingServerProperties();
            CrawlerProps.ProxiesRotation = ProxiesRotationType.SequenciveRotation;
            CrawlerProps.HTTPWebRequestProxiesList = new List<DataCrawlingWebProxy>();
            CrawlerProps.HTTPWebRequestProxiesList.Add(new DataCrawlingWebProxy("127.0.0.1", 601));
            CrawlerProps.HTTPWebRequestProxiesList.Add(new DataCrawlingWebProxy("192.168.0.101", 450, true, "UserName", "UserPassword"));

            //3. Create task
            DataExcavatorTask NewTask = new DataExcavatorTask(
                "Task1",
                "https://www.walmart.com",
                "Some description",
                new List<DataGrabbingPattern>(),
                CrawlerProps,
                new GrabbingServerProperties(),
                "C:/Walmart1"
            );
        }
    }

## 6. Extras

Web data scraping is a powerful tool for many tasks. Often, when there are no other options to obtain data from the target site, you only need to use a quality scraper.
However, you should keep in mind that scraping always involves the need to respect copyrights. It may be content rights, image rights or prohibitions on automatic content collection.
Either way, you must assess the potential risks and be ethical about automatic data scraping.

## 7. Contacts

Contact form url: https://data-excavator.com/contact/
Bug report: https://data-excavator.com/bug-report/
E-mail address: info@mail.data-excavator.com