Tutorial on Automated Uploading Files – Web Scraping with Selenium in Python

Use-case:

A python tutorial with example on how to automate uploading files in web scraping projects using two methods. It solves any issue or problem you may be having for any types of HTML or webpages. Specially follow this tutorial if you have problems uploading a file in Web Scraping or if your code is not working. We use selenium web driver (for chrome) in Python for this tutorial.

We provide two different ways to upload files which cover any type of html or webpage and should be functionable for all of them. In this example, we want to upload an image file to an online website (for resize image purpose). We use two methods

1. Using send.keys function in selenium
2. Using pyAutoGui package in python – covering the cases if send.keys is not working.

Video Tutorial for this blog is available here:

I recommend watching this video as well for better clarification of how these below steps works.

Step1: package

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
import time
import pyautogui

Download the webdriver manager for selenium . You can use most of the common web broswer versions. Here I use the chrome web driver for selenium. In order to do that, I need to go this website and download a zip file related to version of the web browser (Chrome) I have in my local computer. Here is the website to download that https://chromedriver.chromium.org/downloads

You end up downloading a zip file, after unzipping that in a path, you will find an exe file which we will use in the code. You do not need to install anything here.

Step2: data input

In this example, we want to upload an image file in the local path “C:\cover.png” to an online website for image resize.(iloveimg.com/resize-image) .

Next, I provide the path to the chrome driver manager exe file I downloaded earlier as in below:

# driver = webdriver.Firefox()
driver = webdriver.Chrome(executable_path='chromedriver.exe')
# for selenium4 do this ( if above line gives error) run the next two lines:
## service = Service(executable_path='C:\Program Files\Chrome Driver\chromedriver.exe')
## driver = webdriver.Chrome(service=service)

driver.implicitly_wait(15)
driver.get('https://www.iloveimg.com/resize-image')

After this, a web browser window will be opened that is automatically guided by this python code.

Step3: locate the xpath to the element in web page for uploading files

After finding the Xpath for this element, here how you add it in a line of code:

upload_file = driver.find_element(By.XPATH,'//*[@id="pickfiles"]')

Step4: file upload using 1) send.keys 2) pyautogui

For many dynamic webpages or for different reasons, you may find that the common send.keys approach from selenium is not working for uploading files. Hence, you have an alternative method here in the last four line of code, where we mimic the behavior of a human user and actually automate putting the file path address of our files in the local machine to the file browser window as in below:

# upload_file.send_keys(r"C:\cover.png")
# trying to import the file using the file browser function, as send_keys function didn't work
upload_file.click()
time.sleep(3)
pyautogui.typewrite(r'C:\cover.png')
pyautogui.press('enter')

Related Links

• code solution script:

After downloading, change the format of this file from .txt to .ipynb to open it as a notebook file.

Check out these related tutorial in your convenient time:

• For python related tutorials, see this playlist of video tutorials: https://www.youtube.com/playlist?list=PL_b86y-oyLzAQOf0W7MbCXYxEkv27gPN6

• a playlist of web scraping related video tutorials: https://www.youtube.com/watch?v=RLnPN4HE-Qs&
  list=PL_b86y-oyLzDp2EBX-k2bjIBW9SgH3sxT