录制 Kinect 流以供以后播放的最可靠方法是什么?

发布于 2024-11-16 20:03:37 字数 219 浏览 2 评论 0原文

我一直在使用Processing 和Cinder 来动态修改Kinect 输入。但是,我还想记录完整的流(深度+颜色+加速度计值,以及其中的其他内容)。我正在录音,这样我就可以在同一材料上尝试不同的效果/处理。

因为我还在学习 Cinder,而且Processing 非常慢/滞后,所以我很难找到有关捕获流的策略的建议 - 任何东西(最好是在 Cinder、oF 或Processing 中)都会非常有帮助。

I have been working with Processing and Cinder to modify Kinect input on the fly. However, I would also like to record the full stream (depth+color+accelerometer values, and whatever else is in there). I'm recording so I can try out different effects/treatments on the same material.

Because I am still just learning Cinder and Processing is quite slow/laggy, I have had trouble finding advice on a strategy for capturing the stream - anything (preferably in Cinder, oF, or Processing) would be really helpful.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(1

人心善变 2024-11-23 20:03:37

我尝试过Processing 和OpenFrameworks。显示两个图像(深度和颜色)时处理速度较慢。 OpenFrameworks 在将数据写入磁盘时会变慢一点,但基本方法如下:

  1. 设置 Openframeworks(打开并编译任何示例以确保您已启动并运行)
  2. 下载 ofxKinect 插件 并复制示例项目如 github 上所述。
  3. 一旦您运行了 OF 和 ofxKinect 示例,只需添加一些变量来保存数据:

在这个基本设置中,我创建了几个 ofImage 实例和一个用于切换保存的布尔值。在示例中,深度和 RGB 缓冲区保存到 ofxCvGrayscaleImage 实例中,但我还没有充分使用 OF 和 OpenCV,不知道如何执行像将图像保存到磁盘这样简单的操作,这就是为什么我使用了两个 ofImage 实例。

我不知道您对Processing、OF、Cinder 的适应程度如何,因此,为了论证起见,我假设您知道自己已经掌握了Processing,但您仍在处理C++。

OF 与Processing 非常相似,但有一些区别:

  1. 在Processing 中,您有变量声明,并且它们在同一文件中使用。在 OF 中,您有一个 .h 文件(在其中声明变量)和 .cpp 文件(在其中初始化和使用变量)。
  2. 在Processing中,您有setup()(初始化变量)和draw()(更新变量并绘制到屏幕)方法,而在OF中,您有setup()(与Processing中相同)、update()(仅更新变量,没有任何内容) Visual)和draw()(使用更新的值绘制到屏幕)
  3. 在处理图像时,由于您使用C++进行编码,因此需要首先分配内存,这与具有内存管理功能的Processing/Java相反。

还有更多差异,我不会在这里详细说明。请查看 wiki 上的 OF for Processing Users

回到 exampleKinect 示例,请参见此处我的基本设置:

.h 文件:

#pragma once

#include "ofMain.h"
#include "ofxOpenCv.h"
#include "ofxKinect.h"

class testApp : public ofBaseApp {
    public:

        void setup();
        void update();
        void draw();
        void exit();

        void drawPointCloud();

        void keyPressed  (int key);
        void mouseMoved(int x, int y );
        void mouseDragged(int x, int y, int button);
        void mousePressed(int x, int y, int button);
        void mouseReleased(int x, int y, int button);
        void windowResized(int w, int h);

        ofxKinect kinect;

        ofxCvColorImage     colorImg;

        ofxCvGrayscaleImage     grayImage;
        ofxCvGrayscaleImage     grayThresh;
        ofxCvGrayscaleImage     grayThreshFar;

        ofxCvContourFinder  contourFinder;

        ofImage             colorData;//to save RGB data to disk
        ofImage             grayData;//to save depth data to disk 

        bool                bThreshWithOpenCV;
        bool                drawPC;
        bool                saveData;//save to disk toggle

        int                 nearThreshold;
        int                 farThreshold;

        int                 angle;

        int                 pointCloudRotationY;
        int                 saveCount;//counter used for naming 'frames'
};

和 .cpp 文件:

#include "testApp.h"


//--------------------------------------------------------------
void testApp::setup() {
    //kinect.init(true);  //shows infrared image
    kinect.init();
    kinect.setVerbose(true);
    kinect.open();

    colorImg.allocate(kinect.width, kinect.height);
    grayImage.allocate(kinect.width, kinect.height);
    grayThresh.allocate(kinect.width, kinect.height);
    grayThreshFar.allocate(kinect.width, kinect.height);
    //allocate memory for these ofImages which will be saved to disk
    colorData.allocate(kinect.width, kinect.height, OF_IMAGE_COLOR);
    grayData.allocate(kinect.width, kinect.height, OF_IMAGE_GRAYSCALE);

    nearThreshold = 230;
    farThreshold  = 70;
    bThreshWithOpenCV = true;

    ofSetFrameRate(60);

    // zero the tilt on startup
    angle = 0;
    kinect.setCameraTiltAngle(angle);

    // start from the front
    pointCloudRotationY = 180;

    drawPC = false;

    saveCount = 0;//init frame counter
}

//--------------------------------------------------------------
void testApp::update() {
    ofBackground(100, 100, 100);

    kinect.update();
    if(kinect.isFrameNew()) // there is a new frame and we are connected
    {

        grayImage.setFromPixels(kinect.getDepthPixels(), kinect.width, kinect.height);

        if(saveData){
            //if toggled, set depth and rgb pixels to respective ofImage, save to disk and update the 'frame' counter 
            grayData.setFromPixels(kinect.getDepthPixels(), kinect.width, kinect.height,true);
            colorData.setFromPixels(kinect.getCalibratedRGBPixels(), kinect.width, kinect.height,true);
            grayData.saveImage("depth"+ofToString(saveCount)+".png");
            colorData.saveImage("color"+ofToString(saveCount)+".png");
            saveCount++;
        }

        //we do two thresholds - one for the far plane and one for the near plane
        //we then do a cvAnd to get the pixels which are a union of the two thresholds. 
        if( bThreshWithOpenCV ){
            grayThreshFar = grayImage;
            grayThresh = grayImage;
            grayThresh.threshold(nearThreshold, true);
            grayThreshFar.threshold(farThreshold);
            cvAnd(grayThresh.getCvImage(), grayThreshFar.getCvImage(), grayImage.getCvImage(), NULL);
        }else{

            //or we do it ourselves - show people how they can work with the pixels

            unsigned char * pix = grayImage.getPixels();
            int numPixels = grayImage.getWidth() * grayImage.getHeight();

            for(int i = 0; i < numPixels; i++){
                if( pix[i] < nearThreshold && pix[i] > farThreshold ){
                    pix[i] = 255;
                }else{
                    pix[i] = 0;
                }
            }
        }

        //update the cv image
        grayImage.flagImageChanged();

        // find contours which are between the size of 20 pixels and 1/3 the w*h pixels.
        // also, find holes is set to true so we will get interior contours as well....
        contourFinder.findContours(grayImage, 10, (kinect.width*kinect.height)/2, 20, false);
    }
}

//--------------------------------------------------------------
void testApp::draw() {
    ofSetColor(255, 255, 255);
    if(drawPC){
        ofPushMatrix();
        ofTranslate(420, 320);
        // we need a proper camera class
        drawPointCloud();
        ofPopMatrix();
    }else{
        kinect.drawDepth(10, 10, 400, 300);
        kinect.draw(420, 10, 400, 300);

        grayImage.draw(10, 320, 400, 300);
        contourFinder.draw(10, 320, 400, 300);
    }


    ofSetColor(255, 255, 255);
    stringstream reportStream;
    reportStream << "accel is: " << ofToString(kinect.getMksAccel().x, 2) << " / "
                                 << ofToString(kinect.getMksAccel().y, 2) << " / " 
                                 << ofToString(kinect.getMksAccel().z, 2) << endl
                 << "press p to switch between images and point cloud, rotate the point cloud with the mouse" << endl
                 << "using opencv threshold = " << bThreshWithOpenCV <<" (press spacebar)" << endl
                 << "set near threshold " << nearThreshold << " (press: + -)" << endl
                 << "set far threshold " << farThreshold << " (press: < >) num blobs found " << contourFinder.nBlobs
                    << ", fps: " << ofGetFrameRate() << endl
                 << "press c to close the connection and o to open it again, connection is: " << kinect.isConnected() << endl
                 << "press s to toggle saving depth and color data. currently saving:  " << saveData << endl
                 << "press UP and DOWN to change the tilt angle: " << angle << " degrees";
    ofDrawBitmapString(reportStream.str(),20,656);
}

void testApp::drawPointCloud() {
    ofScale(400, 400, 400);
    int w = 640;
    int h = 480;
    ofRotateY(pointCloudRotationY);
    float* distancePixels = kinect.getDistancePixels();
    glBegin(GL_POINTS);
    int step = 2;
    for(int y = 0; y < h; y += step) {
        for(int x = 0; x < w; x += step) {
            ofPoint cur = kinect.getWorldCoordinateFor(x, y);
            ofColor color = kinect.getCalibratedColorAt(x,y);
            glColor3ub((unsigned char)color.r,(unsigned char)color.g,(unsigned char)color.b);
            glVertex3f(cur.x, cur.y, cur.z);
        }
    }
    glEnd();
}

//--------------------------------------------------------------
void testApp::exit() {
    kinect.setCameraTiltAngle(0); // zero the tilt on exit
    kinect.close();
}

//--------------------------------------------------------------
void testApp::keyPressed (int key) {
    switch (key) {
        case ' ':
            bThreshWithOpenCV = !bThreshWithOpenCV;
        break;
        case'p':
            drawPC = !drawPC;
            break;

        case '>':
        case '.':
            farThreshold ++;
            if (farThreshold > 255) farThreshold = 255;
            break;
        case '<':       
        case ',':       
            farThreshold --;
            if (farThreshold < 0) farThreshold = 0;
            break;

        case '+':
        case '=':
            nearThreshold ++;
            if (nearThreshold > 255) nearThreshold = 255;
            break;
        case '-':       
            nearThreshold --;
            if (nearThreshold < 0) nearThreshold = 0;
            break;
        case 'w':
            kinect.enableDepthNearValueWhite(!kinect.isDepthNearValueWhite());
            break;
        case 'o':
            kinect.setCameraTiltAngle(angle);   // go back to prev tilt
            kinect.open();
            break;
        case 'c':
            kinect.setCameraTiltAngle(0);       // zero the tilt
            kinect.close();
            break;
        case 's'://s to toggle saving data
            saveData = !saveData;
            break;

        case OF_KEY_UP:
            angle++;
            if(angle>30) angle=30;
            kinect.setCameraTiltAngle(angle);
            break;

        case OF_KEY_DOWN:
            angle--;
            if(angle<-30) angle=-30;
            kinect.setCameraTiltAngle(angle);
            break;
    }
}

//--------------------------------------------------------------
void testApp::mouseMoved(int x, int y) {
    pointCloudRotationY = x;
}

//--------------------------------------------------------------
void testApp::mouseDragged(int x, int y, int button)
{}

//--------------------------------------------------------------
void testApp::mousePressed(int x, int y, int button)
{}

//--------------------------------------------------------------
void testApp::mouseReleased(int x, int y, int button)
{}

//--------------------------------------------------------------
void testApp::windowResized(int w, int h)
{}

这是一个非常基本的设置。随意修改(在保存的数据中添加倾斜角度等)
我很确定有方法可以提高这种速度(例如,不要更新 ofxCvGrayscaleImage 实例,并且在保存时不要将图像绘制到屏幕上,或者堆叠几帧并按间隔写入它们,而不是在每一帧上写入,等等。 )

祝你好运

I've tried both Processing and OpenFrameworks. Processing is slower when displaying both images (depth and colour). OpenFrameworks slows a bit while writing the data to disk, but here's the basic approach:

  1. Setup Openframeworks (open and compile any sample to make sure you're up and running)
  2. Download the ofxKinect addon and copy the example project as described on github.
  3. Once you've got OF and the ofxKinect example running, it's just a matter of adding a few variable to save your data:

In this basic setup, I've created a couple of ofImage instances and a boolean to toggle saving. In the example the depth and RGB buffers are saved into ofxCvGrayscaleImage instances, but I haven't used OF and OpenCV enough to know how to do something as simple as saving an image to disk, which is why I've used two ofImage instances.

I don't know how comfortable you are with Processing, OF, Cinder, so, for arguments' sake I'll assume you know you're way around Processing, but you're still tackling C++.

OF is pretty similar to Processing, but there are a few differences:

  1. In Processing you have variables declaration and they're usage in the same file. In OF you've got a .h file where you declare you're variables and the .cpp file where you initialize and use your variables.
  2. In Processing you have the setup()(initialize variables) and draw()(update variables and draw to screen) methods, while in OF you have setup() (same as in Processing), update() (update variables only, nothing visual) and draw() (draw to screen using updated values)
  3. When working with images, you since you're coding in C++, you need to allocate memory first, as opposed to Processing/Java where you have memory management.

There's more differences that I won'te detail here. Do check out OF for Processing Users on the wiki

Back to the exampleKinect example, here my basic setup:

.h file:

#pragma once

#include "ofMain.h"
#include "ofxOpenCv.h"
#include "ofxKinect.h"

class testApp : public ofBaseApp {
    public:

        void setup();
        void update();
        void draw();
        void exit();

        void drawPointCloud();

        void keyPressed  (int key);
        void mouseMoved(int x, int y );
        void mouseDragged(int x, int y, int button);
        void mousePressed(int x, int y, int button);
        void mouseReleased(int x, int y, int button);
        void windowResized(int w, int h);

        ofxKinect kinect;

        ofxCvColorImage     colorImg;

        ofxCvGrayscaleImage     grayImage;
        ofxCvGrayscaleImage     grayThresh;
        ofxCvGrayscaleImage     grayThreshFar;

        ofxCvContourFinder  contourFinder;

        ofImage             colorData;//to save RGB data to disk
        ofImage             grayData;//to save depth data to disk 

        bool                bThreshWithOpenCV;
        bool                drawPC;
        bool                saveData;//save to disk toggle

        int                 nearThreshold;
        int                 farThreshold;

        int                 angle;

        int                 pointCloudRotationY;
        int                 saveCount;//counter used for naming 'frames'
};

and the .cpp file:

#include "testApp.h"


//--------------------------------------------------------------
void testApp::setup() {
    //kinect.init(true);  //shows infrared image
    kinect.init();
    kinect.setVerbose(true);
    kinect.open();

    colorImg.allocate(kinect.width, kinect.height);
    grayImage.allocate(kinect.width, kinect.height);
    grayThresh.allocate(kinect.width, kinect.height);
    grayThreshFar.allocate(kinect.width, kinect.height);
    //allocate memory for these ofImages which will be saved to disk
    colorData.allocate(kinect.width, kinect.height, OF_IMAGE_COLOR);
    grayData.allocate(kinect.width, kinect.height, OF_IMAGE_GRAYSCALE);

    nearThreshold = 230;
    farThreshold  = 70;
    bThreshWithOpenCV = true;

    ofSetFrameRate(60);

    // zero the tilt on startup
    angle = 0;
    kinect.setCameraTiltAngle(angle);

    // start from the front
    pointCloudRotationY = 180;

    drawPC = false;

    saveCount = 0;//init frame counter
}

//--------------------------------------------------------------
void testApp::update() {
    ofBackground(100, 100, 100);

    kinect.update();
    if(kinect.isFrameNew()) // there is a new frame and we are connected
    {

        grayImage.setFromPixels(kinect.getDepthPixels(), kinect.width, kinect.height);

        if(saveData){
            //if toggled, set depth and rgb pixels to respective ofImage, save to disk and update the 'frame' counter 
            grayData.setFromPixels(kinect.getDepthPixels(), kinect.width, kinect.height,true);
            colorData.setFromPixels(kinect.getCalibratedRGBPixels(), kinect.width, kinect.height,true);
            grayData.saveImage("depth"+ofToString(saveCount)+".png");
            colorData.saveImage("color"+ofToString(saveCount)+".png");
            saveCount++;
        }

        //we do two thresholds - one for the far plane and one for the near plane
        //we then do a cvAnd to get the pixels which are a union of the two thresholds. 
        if( bThreshWithOpenCV ){
            grayThreshFar = grayImage;
            grayThresh = grayImage;
            grayThresh.threshold(nearThreshold, true);
            grayThreshFar.threshold(farThreshold);
            cvAnd(grayThresh.getCvImage(), grayThreshFar.getCvImage(), grayImage.getCvImage(), NULL);
        }else{

            //or we do it ourselves - show people how they can work with the pixels

            unsigned char * pix = grayImage.getPixels();
            int numPixels = grayImage.getWidth() * grayImage.getHeight();

            for(int i = 0; i < numPixels; i++){
                if( pix[i] < nearThreshold && pix[i] > farThreshold ){
                    pix[i] = 255;
                }else{
                    pix[i] = 0;
                }
            }
        }

        //update the cv image
        grayImage.flagImageChanged();

        // find contours which are between the size of 20 pixels and 1/3 the w*h pixels.
        // also, find holes is set to true so we will get interior contours as well....
        contourFinder.findContours(grayImage, 10, (kinect.width*kinect.height)/2, 20, false);
    }
}

//--------------------------------------------------------------
void testApp::draw() {
    ofSetColor(255, 255, 255);
    if(drawPC){
        ofPushMatrix();
        ofTranslate(420, 320);
        // we need a proper camera class
        drawPointCloud();
        ofPopMatrix();
    }else{
        kinect.drawDepth(10, 10, 400, 300);
        kinect.draw(420, 10, 400, 300);

        grayImage.draw(10, 320, 400, 300);
        contourFinder.draw(10, 320, 400, 300);
    }


    ofSetColor(255, 255, 255);
    stringstream reportStream;
    reportStream << "accel is: " << ofToString(kinect.getMksAccel().x, 2) << " / "
                                 << ofToString(kinect.getMksAccel().y, 2) << " / " 
                                 << ofToString(kinect.getMksAccel().z, 2) << endl
                 << "press p to switch between images and point cloud, rotate the point cloud with the mouse" << endl
                 << "using opencv threshold = " << bThreshWithOpenCV <<" (press spacebar)" << endl
                 << "set near threshold " << nearThreshold << " (press: + -)" << endl
                 << "set far threshold " << farThreshold << " (press: < >) num blobs found " << contourFinder.nBlobs
                    << ", fps: " << ofGetFrameRate() << endl
                 << "press c to close the connection and o to open it again, connection is: " << kinect.isConnected() << endl
                 << "press s to toggle saving depth and color data. currently saving:  " << saveData << endl
                 << "press UP and DOWN to change the tilt angle: " << angle << " degrees";
    ofDrawBitmapString(reportStream.str(),20,656);
}

void testApp::drawPointCloud() {
    ofScale(400, 400, 400);
    int w = 640;
    int h = 480;
    ofRotateY(pointCloudRotationY);
    float* distancePixels = kinect.getDistancePixels();
    glBegin(GL_POINTS);
    int step = 2;
    for(int y = 0; y < h; y += step) {
        for(int x = 0; x < w; x += step) {
            ofPoint cur = kinect.getWorldCoordinateFor(x, y);
            ofColor color = kinect.getCalibratedColorAt(x,y);
            glColor3ub((unsigned char)color.r,(unsigned char)color.g,(unsigned char)color.b);
            glVertex3f(cur.x, cur.y, cur.z);
        }
    }
    glEnd();
}

//--------------------------------------------------------------
void testApp::exit() {
    kinect.setCameraTiltAngle(0); // zero the tilt on exit
    kinect.close();
}

//--------------------------------------------------------------
void testApp::keyPressed (int key) {
    switch (key) {
        case ' ':
            bThreshWithOpenCV = !bThreshWithOpenCV;
        break;
        case'p':
            drawPC = !drawPC;
            break;

        case '>':
        case '.':
            farThreshold ++;
            if (farThreshold > 255) farThreshold = 255;
            break;
        case '<':       
        case ',':       
            farThreshold --;
            if (farThreshold < 0) farThreshold = 0;
            break;

        case '+':
        case '=':
            nearThreshold ++;
            if (nearThreshold > 255) nearThreshold = 255;
            break;
        case '-':       
            nearThreshold --;
            if (nearThreshold < 0) nearThreshold = 0;
            break;
        case 'w':
            kinect.enableDepthNearValueWhite(!kinect.isDepthNearValueWhite());
            break;
        case 'o':
            kinect.setCameraTiltAngle(angle);   // go back to prev tilt
            kinect.open();
            break;
        case 'c':
            kinect.setCameraTiltAngle(0);       // zero the tilt
            kinect.close();
            break;
        case 's'://s to toggle saving data
            saveData = !saveData;
            break;

        case OF_KEY_UP:
            angle++;
            if(angle>30) angle=30;
            kinect.setCameraTiltAngle(angle);
            break;

        case OF_KEY_DOWN:
            angle--;
            if(angle<-30) angle=-30;
            kinect.setCameraTiltAngle(angle);
            break;
    }
}

//--------------------------------------------------------------
void testApp::mouseMoved(int x, int y) {
    pointCloudRotationY = x;
}

//--------------------------------------------------------------
void testApp::mouseDragged(int x, int y, int button)
{}

//--------------------------------------------------------------
void testApp::mousePressed(int x, int y, int button)
{}

//--------------------------------------------------------------
void testApp::mouseReleased(int x, int y, int button)
{}

//--------------------------------------------------------------
void testApp::windowResized(int w, int h)
{}

This is a very basic setup. Feel free to modify (add tilt angle to the saved data, etc.)
I'm pretty sure there are ways to improve this speedwise (e.g. don't update ofxCvGrayscaleImage instances and don't draw images to screen while saving, or stack a few frames and write them at interval as opposed to on every frame, etc.)

Goodluck

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文