Animator Tutorial: Difference between revisions

From FACEGOOD
Jump to navigation Jump to search
(Created blank page)
 
No edit summary
Line 1: Line 1:
=== Overall Workflow ===


# Capturing with Capture Studio, make sure you have a calibration video and a performance video(where it should have a natural pose inside it)
# Set up Python and your system environment
# Camera Calibration
# Stereo Camera Video Data Processing
# Import data to Unreal
# Produce Identity Data and solve anim data
=== EPIC Reference Tutorial ===
You can also access those tutorials to have better knowledge of the whole pipeline.
# <nowiki>https://www.youtube.com/watch?v=qPhn28Jk3Mo</nowiki>
# <nowiki>https://www.youtube.com/watch?v=d3k9rfA9xjs</nowiki>
# <nowiki>https://www.youtube.com/watch?v=-2DH4nX7wT8</nowiki>
# <nowiki>https://www.youtube.com/watch?v=Nkb4DEoZ_NY</nowiki>
# <nowiki>https://dev.epicgames.com/documentation/en-us/metahuman/metahuman-for-unreal-engine/capture-guidelines#usingastereoheadmount</nowiki>
# <nowiki>https://dev.epicgames.com/documentation/en-us/metahuman/metahuman-for-unreal-engine/stereo-camera-calibration-and-tools</nowiki>
=== Set up Python and your system environment ===
Before get started, please read the Preparation Guidelines: [https://dev.epicgames.com/documentation/en-us/metahuman/metahuman-for-unreal-engine/stereo-camera-calibration-and-tools Stereo Camera Calibration and Tools]
'''NOTES:'''
* Require proficiency in using Python, ffmpeg, and Windows system environment settings
* Install ffmpeg (optional,It depends on whether your video has timecode.)
* Click this [https://d1iv7db44yhgxn.cloudfront.net/documentation/attachments/fa394bb1-505d-4c53-9fb2-faae9a36797e/stereocapturetools.zip link] to download the Stereo Capture Tools  to your computer. Unzip the file and open the folder.
==== Install FFmpeg ====
The bundled Python scripts require FFmpeg to be installed and available on the system path. You can download FFmpeg from [https://ffmpeg.org/download.html ffmpeg.org].
Once installed make sure to add the folder where the '''ffmpeg.exe''' binary resides to the system path. You can do this via '''Windows Settings > Environment Variables''', and then adding it as a new entry to your path.
==== Install Python ====
  To run the bundled Python scripts, install '''Python 3.9.7''' or above. Once installed, add Python to your user's path via the checkbox in the installer. You can download Python from [https://www.python.org/downloads/windows/ python.org].In order to better control different versions of python, it is recommended to use [https://docs.anaconda.com/anaconda/install/index.html anaconda].
==== Install Python Dependencies ====
* Locate the requirements.txt file within the '''stereo_capture_tools-0.1.9''' folder.
* Using "conda activate base" in your cmd to activate the python environment, you can also type "conda env list" to check the python environment that you already installed. And then change path to requirements.txt in cmd.
* And then type "pip install -r requirements.txt" to install the python environment.
=== Capturing ===
==== Capture Guidelines ====
Capture Guidelines include details on camera settings, lighting conditions, framing, and other technical aspects necessary to achieve the desired quality and consistency in the captured footage. Following capture guidelines ensures that the footage meets the required standards for further processing and production.
* Specific calibration footage should be taken at the start and the end of a capture session to bookend the shoot.
* During calibration capture, you are attempting to ''paint the space'' with the calibration board. For each capture, the board should be moved across the field of view, including movements of up, down, left, right, roll, and pitch, '''in the plane where the performer’s face will be or was positioned'''. The video should be approximately 20 seconds long.
* The following images show examples of poor calibration techniques.
<u>Hold the head of the helmet with your left hand, hold the calibration plate on your forehead, and move within the  '''performer's face'''  area.</u>
=== Data Processing ===
==== Environment Confirm ====
Type "conda env list" in your CMD to check the python environment.
Type "conda activate base" in your CMD to activate the python environment.
'''Please ensure that all data is accompanied by timecode information.'''
* If the recorded video does not contain timecode information, you can manually add timecode to the video using the following command.
<code>ffmpeg -i performance\bot.mp4 -c copy -y -timecode 00:00:00:00 botT.mp4</code>
==== Data Structure ====
  To demonstrate how to prepare sample footage, you will run through a quick example. Here we assume that you have some calibration footage and two actor performances. One actor performance will be used for identity creation and the other as an actual performance.
  In each of these cases, there are two video files ('''bot and top''') with an additional audio file (audio.wav) for the performances. Or, more visually:
<code>+---example_data</code>
<code>|   +---identity</code>
<code>|   |       audio.wav(optional)</code>
<code>|   |       bot.mov</code>
<code>|   |       top.mov</code>
<code>|   |</code>
<code>|   ---performance</code>
<code>|           audio.wav(optional)</code>
<code>|           bot.mov</code>
<code>|           top.mov</code>
<code>|</code>
<code>+---calibration</code>
<code>|        bot.mov</code>
<code>|        top.mov</code>
==== Process the Calibration data (converting the footage to image sequences) ====
This command will convert video into image frames for import into Unreal for use
<code>python mh_ingest_convert.py</code>
<code>bot c:\MyData\example_data\calibration\bot.mov</code>
<code>top c:\MyData\example_data\calibration\top.mov</code>
<code>c:\MyData\example_data_prepared\calibration</code>
'''<u>Enter the command in your cmd without line breaks.</u>'''
If you are using '''Unreal 5.3 or newer version''', you need to identify the output image format
<code>python "C:\Program Files\Epic Games\UE_5.3\Engine\Plugins\Marketplace\MetaHumanTest\Content\StereoCaptureTools\stereo_capture_tools\stereo_capture_tools\mh_ingest_convert.py"</code>
<code>bot c:\MyData\example_data\calibration\bot.mov</code>
<code>top c:\MyData\example_data\calibration\top.mov</code>
<code>png_rgb24</code>
<code>c:\MyData\example_data_prepared\calibration</code>
'''<u>Otherwise, you will get this error. Pick any format from 'png_gray', 'png_rgb24', 'jpg_copy', 'jpg_lossy'</u>'''
==== Process the Identity data (converting the footage to image sequences) ====
Same operation as above.
If you need audio, you can use
<code>python "C:\Program Files\Epic Games\UE_5.3\Engine\Plugins\Marketplace\MetaHumanTest\Content\StereoCaptureTools\stereo_capture_tools\stereo_capture_tools\mh_ingest_convert.py"</code>
<code>--audio-path C:\MyData\example_data\performance\audio.wav</code>
<code>bot c:\MyData\example_data\identity\bot.mov</code>
<code>top c:\MyData\example_data\identity\top.mov</code>
<code>png_rgb24</code>
<code>c:\MyData\example_data_prepared\identity</code>
==== Process the Performance data (converting the footage to image sequences) ====
Same operation as above.

Revision as of 07:10, 14 August 2024

Overall Workflow

  1. Capturing with Capture Studio, make sure you have a calibration video and a performance video(where it should have a natural pose inside it)
  2. Set up Python and your system environment
  3. Camera Calibration
  4. Stereo Camera Video Data Processing
  5. Import data to Unreal
  6. Produce Identity Data and solve anim data

EPIC Reference Tutorial

You can also access those tutorials to have better knowledge of the whole pipeline.

  1. https://www.youtube.com/watch?v=qPhn28Jk3Mo
  2. https://www.youtube.com/watch?v=d3k9rfA9xjs
  3. https://www.youtube.com/watch?v=-2DH4nX7wT8
  4. https://www.youtube.com/watch?v=Nkb4DEoZ_NY
  5. https://dev.epicgames.com/documentation/en-us/metahuman/metahuman-for-unreal-engine/capture-guidelines#usingastereoheadmount
  6. https://dev.epicgames.com/documentation/en-us/metahuman/metahuman-for-unreal-engine/stereo-camera-calibration-and-tools

Set up Python and your system environment

Before get started, please read the Preparation Guidelines: Stereo Camera Calibration and Tools

NOTES:

  • Require proficiency in using Python, ffmpeg, and Windows system environment settings
  • Install ffmpeg (optional,It depends on whether your video has timecode.)
  • Click this link to download the Stereo Capture Tools to your computer. Unzip the file and open the folder.

Install FFmpeg

The bundled Python scripts require FFmpeg to be installed and available on the system path. You can download FFmpeg from ffmpeg.org.

Once installed make sure to add the folder where the ffmpeg.exe binary resides to the system path. You can do this via Windows Settings > Environment Variables, and then adding it as a new entry to your path.

Install Python

  To run the bundled Python scripts, install Python 3.9.7 or above. Once installed, add Python to your user's path via the checkbox in the installer. You can download Python from python.org.In order to better control different versions of python, it is recommended to use anaconda.

Install Python Dependencies

  • Locate the requirements.txt file within the stereo_capture_tools-0.1.9 folder.
  • Using "conda activate base" in your cmd to activate the python environment, you can also type "conda env list" to check the python environment that you already installed. And then change path to requirements.txt in cmd.
  • And then type "pip install -r requirements.txt" to install the python environment.

Capturing

Capture Guidelines

Capture Guidelines include details on camera settings, lighting conditions, framing, and other technical aspects necessary to achieve the desired quality and consistency in the captured footage. Following capture guidelines ensures that the footage meets the required standards for further processing and production.

  • Specific calibration footage should be taken at the start and the end of a capture session to bookend the shoot.
  • During calibration capture, you are attempting to paint the space with the calibration board. For each capture, the board should be moved across the field of view, including movements of up, down, left, right, roll, and pitch, in the plane where the performer’s face will be or was positioned. The video should be approximately 20 seconds long.
  • The following images show examples of poor calibration techniques.

Hold the head of the helmet with your left hand, hold the calibration plate on your forehead, and move within the performer's face area.

Data Processing

Environment Confirm

Type "conda env list" in your CMD to check the python environment. Type "conda activate base" in your CMD to activate the python environment.

Please ensure that all data is accompanied by timecode information.

  • If the recorded video does not contain timecode information, you can manually add timecode to the video using the following command.

ffmpeg -i performance\bot.mp4 -c copy -y -timecode 00:00:00:00 botT.mp4

Data Structure

  To demonstrate how to prepare sample footage, you will run through a quick example. Here we assume that you have some calibration footage and two actor performances. One actor performance will be used for identity creation and the other as an actual performance.

  In each of these cases, there are two video files (bot and top) with an additional audio file (audio.wav) for the performances. Or, more visually:

+---example_data

|   +---identity

|   |       audio.wav(optional)

|   |       bot.mov

|   |       top.mov

|   |

|   ---performance

|           audio.wav(optional)

|           bot.mov

|           top.mov

|

+---calibration

|        bot.mov

|        top.mov

Process the Calibration data (converting the footage to image sequences)

This command will convert video into image frames for import into Unreal for use

python mh_ingest_convert.py

bot c:\MyData\example_data\calibration\bot.mov

top c:\MyData\example_data\calibration\top.mov

c:\MyData\example_data_prepared\calibration

Enter the command in your cmd without line breaks.

If you are using Unreal 5.3 or newer version, you need to identify the output image format

python "C:\Program Files\Epic Games\UE_5.3\Engine\Plugins\Marketplace\MetaHumanTest\Content\StereoCaptureTools\stereo_capture_tools\stereo_capture_tools\mh_ingest_convert.py"

bot c:\MyData\example_data\calibration\bot.mov

top c:\MyData\example_data\calibration\top.mov

png_rgb24

c:\MyData\example_data_prepared\calibration

Otherwise, you will get this error. Pick any format from 'png_gray', 'png_rgb24', 'jpg_copy', 'jpg_lossy'

Process the Identity data (converting the footage to image sequences)

Same operation as above.

If you need audio, you can use

python "C:\Program Files\Epic Games\UE_5.3\Engine\Plugins\Marketplace\MetaHumanTest\Content\StereoCaptureTools\stereo_capture_tools\stereo_capture_tools\mh_ingest_convert.py"

--audio-path C:\MyData\example_data\performance\audio.wav

bot c:\MyData\example_data\identity\bot.mov

top c:\MyData\example_data\identity\top.mov

png_rgb24

c:\MyData\example_data_prepared\identity

Process the Performance data (converting the footage to image sequences)

Same operation as above.