Animator Tutorial
Overall Workflow
- Capturing with Capture Studio, make sure you have a calibration video and a performance video(where it should have a natural pose inside it)
- Set up Python and your system environment
- Camera Calibration
- Stereo Camera Video Data Processing
- Import data to Unreal
- Produce Identity Data and solve anim data
EPIC Reference Tutorial
You can also access those tutorials to have better knowledge of the whole pipeline.
- https://www.youtube.com/watch?v=qPhn28Jk3Mo
- https://www.youtube.com/watch?v=d3k9rfA9xjs
- https://www.youtube.com/watch?v=-2DH4nX7wT8
- https://www.youtube.com/watch?v=Nkb4DEoZ_NY
- https://dev.epicgames.com/documentation/en-us/metahuman/metahuman-for-unreal-engine/capture-guidelines#usingastereoheadmount
- https://dev.epicgames.com/documentation/en-us/metahuman/metahuman-for-unreal-engine/stereo-camera-calibration-and-tools
Set up Python and your system environment
Before get started, please read the Preparation Guidelines: Stereo Camera Calibration and Tools
NOTES:
- Require proficiency in using Python, ffmpeg, and Windows system environment settings
- Install ffmpeg (optional，It depends on whether your video has timecode.)
- Click this link to download the Stereo Capture Tools to your computer. Unzip the file and open the folder.
Install FFmpeg
The bundled Python scripts require FFmpeg to be installed and available on the system path. You can download FFmpeg from ffmpeg.org.
Once installed make sure to add the folder where the ffmpeg.exe binary resides to the system path. You can do this via Windows Settings > Environment Variables, and then adding it as a new entry to your path.
Install Python
To run the bundled Python scripts, install Python 3.9.7 or above. Once installed, add Python to your user's path via the checkbox in the installer. You can download Python from python.org.In order to better control different versions of python, it is recommended to use anaconda.
Install Python Dependencies
- Locate the requirements.txt file within the stereo_capture_tools-0.1.9 folder.
- Using "conda activate base" in your cmd to activate the python environment, you can also type "conda env list" to check the python environment that you already installed. And then change path to requirements.txt in cmd.
- And then type "pip install -r requirements.txt" to install the python environment.
Capturing
Capture Guidelines
Capture Guidelines include details on camera settings, lighting conditions, framing, and other technical aspects necessary to achieve the desired quality and consistency in the captured footage. Following capture guidelines ensures that the footage meets the required standards for further processing and production.
- Specific calibration footage should be taken at the start and the end of a capture session to bookend the shoot.
- During calibration capture, you are attempting to paint the space with the calibration board. For each capture, the board should be moved across the field of view, including movements of up, down, left, right, roll, and pitch, in the plane where the performer’s face will be or was positioned. The video should be approximately 20 seconds long.
- The following images show examples of poor calibration techniques.
Hold the head of the helmet with your left hand, hold the calibration plate on your forehead, and move within the performer's face area.
Data Processing
Environment Confirm
Type "conda env list" in your CMD to check the python environment. Type "conda activate base" in your CMD to activate the python environment.
Please ensure that all data is accompanied by timecode information.
- If the recorded video does not contain timecode information, you can manually add timecode to the video using the following command.
ffmpeg -i performance\bot.mp4 -c copy -y -timecode 00:00:00:00 botT.mp4
Data Structure
To demonstrate how to prepare sample footage, you will run through a quick example. Here we assume that you have some calibration footage and two actor performances. One actor performance will be used for identity creation and the other as an actual performance.
In each of these cases, there are two video files (bot and top) with an additional audio file (audio.wav) for the performances. Or, more visually:
+---example_data
| +---identity
| | audio.wav(optional)
| | bot.mov
| | top.mov
| |
| ---performance
| audio.wav(optional)
| bot.mov
| top.mov
|
+---calibration
| bot.mov
| top.mov
Process the Calibration data (converting the footage to image sequences)
This command will convert video into image frames for import into Unreal for use
python mh_ingest_convert.py
bot c:\MyData\example_data\calibration\bot.mov
top c:\MyData\example_data\calibration\top.mov
c:\MyData\example_data_prepared\calibration
Enter the command in your cmd without line breaks.
If you are using Unreal 5.3 or newer version, you need to identify the output image format
python "C:\Program Files\Epic Games\UE_5.3\Engine\Plugins\Marketplace\MetaHumanTest\Content\StereoCaptureTools\stereo_capture_tools\stereo_capture_tools\mh_ingest_convert.py"
bot c:\MyData\example_data\calibration\bot.mov
top c:\MyData\example_data\calibration\top.mov
png_rgb24
c:\MyData\example_data_prepared\calibration
Otherwise, you will get this error. Pick any format from 'png_gray', 'png_rgb24', 'jpg_copy', 'jpg_lossy'
Process the Identity data (converting the footage to image sequences)
Same operation as above.
If you need audio, you can use
python "C:\Program Files\Epic Games\UE_5.3\Engine\Plugins\Marketplace\MetaHumanTest\Content\StereoCaptureTools\stereo_capture_tools\stereo_capture_tools\mh_ingest_convert.py"
--audio-path C:\MyData\example_data\performance\audio.wav
bot c:\MyData\example_data\identity\bot.mov
top c:\MyData\example_data\identity\top.mov
png_rgb24
c:\MyData\example_data_prepared\identity
Process the Performance data (converting the footage to image sequences)
Same operation as above.