How to delete all rows which contains a specific value or character in a mysql database table

by

On my last posts I had crawled a lot of domain name on a mysql table. And now there is a new problem with specific characters like “:” and port numbers. For example: Now I want to delete all this domain names which contains the port numbers in field “url”. MySQL command for find and … Read more

how to remove duplicate rows in mysql with using temporary table

by

after i get 100 Million of url’s, i saw a few of duplicates on my mysql table. I used the command line because the MySQL Workbench was not usable with minimum system resources. For login on Terminal on Mac OS X with MySQL Community Server version: 8.0.29 – GPL after this command, u need to … Read more

Web Scraping with Scrapy pipeline to add crawled data to MongoDB collection [Tutorial]

by

In this tutorial i want to show you how to add the scraped data from scrapy crawler to a MongoDB database. For this we will use the scrapy crawler pipeline with the correct connection to a localhost server. This tutorial will walk you through these tasks: In this Scrapy project I scrape quotes from https://quotes.toscrape.com/ … Read more

Headless ecommerce open source frameworks list [API, JSON, GraphQL]

by
https://unsplash.com/@mikafinland

headless ecommerce open source framework are very popular on 2022. With newest technologies you can convert or integrate your old ecommerce Shop to a new headless open source ecommerce with API’s and GraphQL. On this post you will get a list of the most popular frameworks with demo links and repositories. A List of open … Read more

crawlers list with github repos – python, go, java, php & co

by

“There are only two hard things in Computer Science: cache invalidation and naming things” Phil Karlton on my last post from 30 March 2022, I started with same crawlers to finding unique hostnames and then collecting them on a mysql database. Example Crawler: dcrawl – searches hostnames from given start url. A free open-source project … Read more

How to build a Search Engine with Laravel & MongoDB and Scrapy [PHP, NoSQL, Python on Linux OS]

by

This article is currently being revised and expanded! Last update on 01.07.2022: getting more then 100 Million of unique domain name on a MySQL Database.Using MeiliSearch for indexing is cool, but after 10 Million Index it will be very slow to indexing new rows.Because of this I am looking for a faster way to indexing … Read more

TOP 10+ open-source e-commerce platforms in 2022

by

comparing open source e-commerce software applications on GitHub and Installation of Cloud Instances like on DigitalOcean, Amazon Web Services(AWS), Google Cloud Platform or on a virtual private server. A comparing of open source e-commerce platforms for unique and clean, and also for fast, modern and e-commerce web applications with awesome interfaces in 2022 PHP, Python … Read more