AUTOMATED APPROACH FOR THE IMPORTING THE NEW PHOTO SET TO PRIVATE PHOTO ALBUM TO MAKE IT MORE SEARCHABLE

This paper focuses on describing several Informational Retrieval (IR) Multimedia Systems that work primarily with private photo albums and it provides comparison of IR Multimedia Systems in terms of functionality. The Informational Retrieval Multimedia systems, used for comparison are selected as the most known and widely used as well. The Informational Retrieval Multimedia systems compared are Mylio, Google Photo, Apple Photo, Synology Moments and digiKam. Moreover, based on these systems comparison, this paper provides the definition of the end-point user needs. Then it provides the identification and the prioritization (based on the author’s experience and opinion) the issues in private information Retrieval multimedia systems to be addressed by any technical solution. The following issues have been addressed as the most important from the end-point user’s perspective: First of all, it is “Not existing EXIF file metadata”.(It is divided into two sub issues: not existing taken date and GEO data for place it was taken at); as second, the linking a photo to an author person entity (Based on camera brand and model given as metadata); And as the last one, linking the set of photos to the predefined type of event, like birthdays, anniversaries, holidays, trips etc.(Based on tagging the photos in semi-automated way). For each issue, this document has provided the technical approaches and program algorithms (in a pseudo code) as well. Also, the using of relevant tools is provided if applicable. Each tool mentioned is described in a few words and given with a reference to read about. This document has also described the human involvement into the process, if a totally automated process is not possible. As a conclusion this paper describes the key points that were identified during the identification and addressing the issues. The conclusion also gives a proposal for a technical solution (that is required to be used during importing the new photo set into existing IR Multimedia systems) with context diagram that represents the user, its devices with multimedia data, the system to import the data from user multimedia devices and Informational Retrieval multimedia system to keep the whole private multimedia and to search the target photo set. Finally, it gives the definition of workflow steps to describe the main activity flow (with human involvement and software as well) to be implemented in a technical solution in future.

This chapter provides a review of the areas of potential intended usage and implementation of Multimedia IR systems (that search through the non-textual information) in different fields. In corporate segment, the target audience might be: -people of different professions who need access to images, like medical professionals searching for medical images or architects requiring the image examples to create the buildings; -video content engineers, when looking for special video segments and movies by their titles; -journalists who work with camera to create multimedia news content; -photographers that provide their services to companies; -car engineers who need photographs and sound of car engines.
Ordinary people also might be interested in such kind of engines. For example, let us calculate the size of a photo album for an average family. Here we assume that an average family consists of 3 persons. Also, let us assume a family owns 2 devices with a camera. On average, one person with a photo device can take 5 photos per day, and thus, the entire family takes 10 photos per day (sometimes it might be 0, but sometimes it might be hundreds). It means that one family can take more than 3 thousand photos annually, and in 10 years -more than 30 thousand of photos. And, if such a family does not have a good tool to search through the 30 thousand of photos of their photo album, the process of finding of target photos might be very difficult.
From an average user's point of view, there are the following requirements to be met by the Multimedia IR system: -it should provide a container to keep different ty pes of multimedia; -it should be possible to represent any kind of multimedia information (photo, video, and audio); -it should provide a simple way to edit the multimedia data; -it should be possible to search through the content quickly and easily; -it should allow filtering of information by several sets of criteria; -relevance feedback shall be implemented; -it should include indexing and cataloguing functions; -it should be possible to find information by example of image; -it should include a function of browsing through the search results. It is clear enough how to keep and represent the multimedia information. However, the question is what kind of search requests are possible in this case.
There are many different technical photo properties/attributes, such as Aperture, Camera make, Camera model, etc. (see the figure below). All of them could be used in the search. In this paper we are focusing only on those properties/attributes that might be used in search by ordinary people. They might be: -Data and time when the photo is taken (e.g.: 2020/04/20 21:22:21); -The people in it (e.g.: my father, friend); -The place where it is taken (for instance, longitude + latitude or "my parents' home"); -Objects in it (forests, pets, cars); -Events description (my father's birthday, a trip to London); -Commentary (any text comments); -Rating or marks (from 0 to 10); -A combination of color, texture (a lot of blue (like the sky)); -Emotions attached to an event (smiles, etc.) -Similar photos.
Let us provide the examples of requests based on photo properties/attributes that people might make when searching for certain photos: -My photos from April 2020; -All photos when I am at home; -My father's anniversaries; -Photos when I was in Italy; -Photos where I am in the forest; -All my cat/dog photos; -Photos with my grandmother from 2000 till 2010 at home; -Photos made by my brother.

Overview and comparison of the private Multimedia IR systems
This chapter provides an overview and comparison of several private Multimedia IR systems.
There are photo organizers that provide the functions similar to private Multimedia IR systems. Therefore, they are chosen as a subject for the overview and comparison. They currently perform the search in photo set by organizing the photos based on their date, location, categories, etc. They suggest to create a folder structure and put the files on it. The folder name can be something like Date_Shoot-Type_Event Name. It can also include a location or any other information the user wants. In such a way the user can create subfolders based on the number of photographers or cameras. The idea is to avoid photos having the same file name in the folder [5]. This [5] provides the list of Photo Organizer for professional photographer and those organizers can be used by a usual user with a big photo album.
There are even relevant online courses available to guide the user how to organize the photos.
The multimedia IR systems are selected for the overview according to the following set of criteria.
One or several of them: -should be online; -should be offline; -shall be free; -shall be paid. The comparison table shows the following aspects: -almost all systems are able to support storage, editing, indexing, browsing and cataloguing the multimedia data; -all systems are able to search using location data and date/time metadata; -all systems are able to define a person using face recognition technology; -only several systems can recognize the objects on the photos; -searching by event is possible only through tagging; -search by feedback and by comments ability is provided by all systems; -all the systems are unable to search by emotions; -only some systems can make the search by a similar photo; -only some systems perform the search by a combination of color or texture; -the search by photo author is successful only by applying tagging. The more detailed comparison of photo organizers is provided by Wikipedia and available here [1].

Identification and prioritization of the issues in IR multimedia systems
This chapter dwells upon the identification and prioritization of the issues found during the overview and compassing the current IR multimedia systems.
The identification and prioritization of the issues was performed based on private needs of the paper's author.

Addressing the issues in private IR multimedia systems
The following three issues with the highest priority are chosen to be addressed in this paper: -It is undefined what to do if metadata does not exist; -It is undefined how to link a photo to an author person entity; -It is undefined how to link a set of photos to the predefined type of event.
The issues shall be addressed in the way mentioned above with a toolset available for a usual user and not for the enterprises.
Snippets of pseudo code or tool screenshots examples will be provided to explain the approach in addressing the issues.
The most automated approach possible should be provided. In case there is a need to perform any manual steps, it should be justified and described as well.

Addressing the issues with not existing metadata
Basing on the metadata inside of photo, the user might perform the search process by date/time and location criteria. So, if location data or date/time does not exist, the user is not able to search in the set of photos by location and date/time requests. Those requests are commonly and widely used, so it is very important to identify if that data does not exist inside the photo.
That metadata is usually kept inside of each file as a header with Exif format. There are several other formats of metadata like IPTC or XMP, but in this paper we are focusing mostly on Exif as the most common metadata header.
The first step is to identify if location or date/ time data is not present. That might be done using the tools like Apache Tika or digiKam [4].
As soon as the photos without the GPS and time/ date taken are identified, it is necessary to assign the correct metadata to the photos in a manual or automated way. Below is the description of the automated way to update GPS and time/date of the photos.
To update the time/date of the photos, the following approach might be used: 1. If assumption about taken date/time was not correct -no photos will be updated and more attempts need to be performed.
As a result of this algorithm, all photos with modification date/time that is inside of the assumed date will get Exif time/date taken that corresponds to the "last modification time". To update Exif metadata we use "jhead" tool that is free to use. That algorithm assumes that the user performing such action is aware of the assumed date when the photo was taken. As soon as date/time taken is attached to the photos, it will be possible to search those photos by date/time criteria, because almost all multimedia search engines take into account the Exif date/time taken attribute.
The next issue is assigning Exif geo data in automatic way. First, it should be mentioned that there are some Internet tools that allow a user to reverse the images to get, for example, the person identification by the photo or to get the geo position based on photo details. For example, they include https:// serpapi.com/google-reverse-image, https://images. google.com/, https://mrisa.mage.me.uk/, etc. So, they look for the similarity in the photos and they return actually. Most likely the researcher will need to pay a fee to use such Internet engines.
The local search engines can also provide the same functionality. However, due to much smaller base, they return less accurate results. Still, they will perform the task for the photos in famous and public places.

Divide the photos into separate folders by
grouping them by the same GPS position. 2. Define the exact geo position using the map service like Google Maps or OpenStreet maps for each photo set inside one folder. 3. For each photo file inside the folder with the same geo position, call the "exiftool" command to set GPS data defined earlier like in step 5 of the previous algorithm. As a result of these algorithm jobs, all photos will get Exif GPS data. Both the algorithms assume that the user performing such action is aware of the assumed place where the photo was taken. As soon as GPS data is attached to the photos, it will be possible to search those photos by position criteria, because almost all multimedia search engines take into account the Exif GPS attributes.

Linking the photo to the author
Usually, during the search in the photo album it is important to find the set of the photos taken by a certain person.
It is assumed that there is a predefined set of authors -persons' list in a form of tag set, and each item of the list should be assigned to each photo. For example, "author\dmytro_nikulin; author\vasia_nikulin; author\petia_nikulin".
It is also assumed that each photo file has Exif "Camera Make" and "Camera Model" attributes assigned.
The string representation of concatenation of "Camera Maker" and "Camera Model" can be a key to link the photo to the author. The following dictionary shall be created to support the linkage: "Camera Maker" + "Camera Model" to "author\ author_name".
Having that dictionary, the following algorithm can be used to assign the photo to the author: 1. Get the photo file; 2. Extract the "Camera Maker" and "Camera Model" attributes; 3. Put the extracted attributes to one string variable CameraMaker&Model; 4. Try to search in the dictionary "Camera Maker" + "Camera Model" to "author\author_ name" by CameraMaker&Model as a dictionary key; 5. If the search is successful -assign current photo file tag to "author\author_name". 6. If the search is not successful -6.1. log the error/warning message with photo file name, or 6.2. show the dialog to user to make it possible to create a new item in the dictionary "Camera Maker" + "Camera Model" to "author\author_name". It might happen that "Camera Maker" + "Camera Model" key is not unique because of different people having the same device model. For example, the dictionary will be as follows: "samsungSM-G965F" -"author\vasia_nikulin"; "samsungSM-G965F" -"author\petia_nikulin". In such cases user might be prompted to select the correct person from the list or a log message might be created to be analyzed after processing.
As a result of that algorithm, all photos will get author tags assigned. The algorithm assumes a user to be involved in the preparation of the "Camera Maker" + "Camera Model" to "author\author_name" dictionary and selection of proper photo author if the algorithm is unable to do it in automatic way. As soon as the author tags are attached to the photos, it will be able to search those photos by author.

Linking the set of photos to an event
The current private IR multimedia systems can perform the search in photo album by the event if appropriate tags are assigned to the photos. For example, a photo file can be assigned the tag such as Figure 4. The example of the photo file with assigned "Camera Maker" and "Camera Model" attributes "event/somebody's anniversary". In this case IR multimedia system can find this photo by user search request: all photos with "event/somebody's anniversary".
First of all, there should be a well-defined tags structure. There are several Internet sources [2], [3] that describe the rules of creating a smart and flexible tags structure.
The following tag structure can be used as a ref- As soon as the tag event structure is defined, the following algorithm to assign event to photos might be executed: 1. Divide the photos in separate folders by grouping by the same event. 2. For each photo file inside the folder with same event, assign the appropriate tag for file -for example, using the digiKam [4] tool.
As a result of this algorithm, all photos will get the event tag assigned. This algorithm assumes that the user is involved in the preparation of the folder structure with splitting the folders by common event and assigning a predefined event tag to the set of photos. As soon as the event tags are assigned to the photos, it will be possible to search those photos by event.

Conclusion
As a result of issues identification and addressing, the following statements can be formulated: -The almost all IR Multimedia systems provide a user a possibility to search by a lot of parameter/attributes and provide a wide functionality for keeping and editing the photo album content. -The metadata is a main information basement to search through the photo album for IR Multimedia systems.
-The search attributes are the subjective ones and might vary from other user needs to others. -The current IR Multimedia systems don't provide a flexible and automated functionality to import the multimedia data into IR Multimedia systems basing on user search needs. -Despite there are a lot of steps that can be automated, still a human involvement is required. So, automation for importing the new photo set into photo album might be highly required feature for IR Multimedia systems as it can identify and address some issues that shall be fixed before user starts searching the data using IR Multimedia system.

The next steps
Basing on the statements above the following user needs are suggested to be covered: -Asking a user for the media attributes he/she is going to use during the photo album search; -Basing on the media attributes that user defined as important one, to keep search attributes configuration; -Basing on the search attributes configuration to identify the gaps inside of multimedia data that is going to imported into the current photo album; -As soon as any gap is identified to automatically cover the missing data; -To involve user interaction as soon as any gap can not be addressed in automatic way. The software for importing the multimedia data to IR Multimedia systems can be developed to cover the user needs described The following picture shows the context diagram for solution proposed: Figure 5. The context diagram of software that imports the multimedia data from user devices to storage of IR multimedia systems.
The work flow, according the figure, is following: 1. The multimedia data from the user devices shall be placed inside of the input folder where the software for import is running on. 2. Once it is done the photo album owner shall start the software for the import. 3. The software for import shall take the multimedia content from input folder and to identify any missing data according to search parameter configuration.
4. As soon as any missing information is detected it shall try to fill it in in automatic way. It shall prompt user if filling the missing data cannot be done in automatic way. 5. Once any missing information is detected and filled in, it shall put the multimedia data into the storage of IR multimedia systems. Storages of IR Multimedia systems are folders where all user photo albums are kept and searchable. They might be Mylio, Google Photo, Apple photo, Synology Moments or digiKam [4].