REPOSITORY OVERVIEWLive repository statistics
★ 8Stars
⑂ 2Forks
◯ 0Open issues
◉ 8Watchers
67/100
OPENREPOHUB HEALTH SIGNALHealthy signals
A transparent discovery signal based on current public GitHub metadata.
Recent activity35% weight
72 Community adoption25% weight
15 Maintenance state20% weight
100 License clarity10% weight
100 Project information10% weight
75 This score does not audit code, security, maintainers, documentation quality, or suitability. Verify the repository and its current documentation before adoption.
README preview
netDigger
netDigger is a web scraping application built using .NET 8.0 and C# WPF in Visual Studio. It collects various types of data and exports them into organized and structured databases. The application features a modern UI design with detailed and comprehensive logging.
Features
- Asynchronous Web Scraping: Efficiently scrape web pages using asynchronous tasks with minimized latency and multi-threading for parallel processing.
- Data Collection: Collects data such as PDFs, CSVs, DOCX, XLS, PPTX, TXT, Images, Videos, JSON, DBSQL, XML, HTML, PHP, JS, Archives, and Miscellaneous files.
- Comprehensive JSON, XML, and HTML Parsing: Utilizes advanced parsing techniques to extract valuable information from JSON, XML, and HTML documents, including finding and processing hidden element data and meta data.
- Database Integration: Organizes scraped URLs into SQLite databases based on their file types.
- Modern UI Design: User-friendly WPF interface with rich text logging.
- Detailed Logging: Comprehensive log messages with timestamps, log levels, and thread IDs.
- Export Options: Export scraped data to database files, CSV, and TXT formats.
- Multi OS Support: Compatible with Windows x64/x86/ARM, Linux and MacOS.
Technologies Used
- .NET 8.0
- C#
- WPF (Windows Presentation Foundation)
- AngleSharp for HTML parsing
- PuppeteerSharp: A headless browser automation library for .NET.
- Newtonsoft.Json (Json.NET): A popular library for working with JSON in .NET.
- SQLite for database management
- Concurrent Collections for thread-safe operations
Prerequisites
- .NET 8.0 Desktop Runtime or SDK Framework.
- Visual Studio 2022. (In case you want to build it yourself).
Installation
-
Clone the repository:
git clone https://github.com/your-username/netDigger.git
cd netDigger
-
Open the solution file (netDigger.sln) in Visual Studio.
-
Build the project:
Select Build > Build Solution.
Run the application:
Select Debug > Start Debugging or press F5.
Contribution
- Fork the repository.
- Create a new branch (git checkout -b feature-branch).
- Commit your changes (git commit -m 'Add new feature').
- Push to the branch (git push origin feature-branch).
- Create a new Pull Request.
License
This project is licensed under the GNU Affero General Public License v3.0. See the LICENSE file for more details.
Support Me
If you find RepoUp useful, consider supporting me by:
- Starring the repository on GitHub
- Sharing the tool with others
- Providing feedback and suggestions
- Follow me for more :)
For any issues or feature requests, please open an issue on GitHub. Happy coding!
ALGORITHMICALLY RELATEDSimilar Open-Source Projects
Selected from shared topics, language and repository description—not editorial ratings.
Pizza ordering application with database connection, stock, member/admin login Visual Studio 2022 .Net Framework 4.7.2 Windows Application with C# MySQL-Workbench 8.0 XAMPP Control Panel 3.3 Installed Nuget> MySql.Data + MySql.Data.EntityFrameworkCore
59/100 healthRecently updatedActive repository
C#No license#crud#csharp#desktop-application#dotnet
⑂ 0 forks◯ 0 issuesUpdated 16 days ago