整理 | 夕颜
出品 | CSDN（ID:CSDNnews）
今天，一个名为 Real-Time-Person-Removal（实时人物去除）项目在GitHub上火了，登上近日GitHub Trending第一，目前已经获得1.8k star。
这个项目的神奇之处在于，只需要在网络浏览器中使用JavaScript，用200多行TensorFlow.js代码，就可以实时让视频画面中的人物对象从复杂的背景中凭空消失！
这虽然不能让你在现实生活中像哈利·波特一样隐身的梦想成真，但至少在视频、动画里可以体验一把隐身的快感！
首先奉上GitHub地址：https://github.com/jasonmayes/Real-Time-Person-Removal
这个项目能干啥？
本项目的作者@jasonmayes（Jason Mayes）是谷歌的一名资深开发者，是机器智能研究和高级开发的倡导者，作为一名TensorFlow.js专家，他拥有超过15年使用新技术开发创新Web解决方案的经验。
他在项目介绍中表示，这段代码的目的在于随着时间的推移学习视频背景的构成，让作者可以尝试从背景中移除任何人物，而所有效果都是使用TensorFlow.js在浏览器中实时实现的。
但同时作者表示，这只是一个实验，并非在所有情况下都是完美的。
消失的人
废话不多说，上代码！
可能有人会觉得在复杂的背景下实现“隐身”是很复杂的吧，而且还是实时的，但实际上实现这样的效果却只需要200多行JS代码：
1/**
2*@License
3*Copyright2018GoogleLLC.AllRightsReserved.
4*LicensedundertheApacheLicense,Version2.0(the"License");
5*youmaynotusethisfileexceptincompliancewiththeLicense.
6*YoumayobtainacopyoftheLicenseat
7*
8*http://www.apache.org/licenses/LICENSE-2.0
9*
10*Unlessrequiredbyapplicablelaworagreedtoinwriting,software
11*distributedundertheLicenseisdistributedonan"ASIS"BASIS,
12*WITHOUTWARRANTIESORCONDITIONSOFANYKIND,eitherexpressorimplied.
13*SeetheLicenseforthespecificlanguagegoverningpermissionsand
14*limitationsundertheLicense.
15*=============================================================================
16*/
17
18/********************************************************************
19*Real-Time-Person-RemovalCreatedbyJasonMayes2020.
20*
21*GetlatestcodeonmyGithub:
22*https://github.com/jasonmayes/Real-Time-Person-Removal
23*
24*Gotquestions?Reachouttomeonsocial:
25*Twitter:@jason_mayes
26*LinkedIn:https://www.linkedin.com/in/creativetech
27********************************************************************/
28
29constvideo=document.getElementById('webcam');
30constliveView=document.getElementById('liveView');
31constdemosSection=document.getElementById('demos');
32constDEBUG=false;
33
34//Anobjecttoconfigureparameterstosetforthebodypixmodel.
35//Seegithubdocsforexplanations.
36constbodyPixProperties={
37architecture:'MobileNetV1',
38outputStride:16,
39multiplier:0.75,
40quantBytes:4
41};
42
43//Anobjecttoconfigureparametersfordetection.Ihaveraised
44//thesegmentationthresholdto90%confidencetoreducethe
45//numberoffalsepositives.
46constsegmentationProperties={
47flipHorizontal:false,
48internalResolution:'high',
49segmentationThreshold:0.9
50};
51
52//Mustbeeven.Thesizeofsquarewewishtosearchforbodyparts.
53//Thisisthesmallestareathatwillrender/notrenderdependingon
54//ifabodypartisfoundinthatsquare.
55constSEARCH_RADIUS=300;
56constSEARCH_OFFSET=SEARCH_RADIUS/2;
57
58
59//RESOLUTION_MINshouldbesmallerthanSEARCHRADIUS.About10xsmallerseemsto
60//workwell.Effectsoverlapinsearchspacetocleanupbodyoverspillforthings
61//thatwerenotclassifiedasbodybutinfactwere.
62constRESOLUTION_MIN=20;
63
64
65//Renderreturnedsegmentationdatatoagivencanvascontext.
66functionprocessSegmentation(canvas,segmentation){
67varctx=canvas.getContext('2d');
68
69//Getdatafromouroverlaycanvaswhichisattemptingtoestimatebackground.
70varimageData=ctx.getImageData(0,0,canvas.width,canvas.height);
71vardata=imageData.data;
72
73//Getdatafromthelivewebcamviewwhichhasalldata.
74varliveData=videoRenderCanvasCtx.getImageData(0,0,canvas.width,canvas.height);
75vardataL=liveData.data;
76
77//Nowloopthroughandseeifpixelscontainhumanparts.Ifnot,update
78//backgoundunderstandingwithnewdata.
79for(letx=RESOLUTION_MIN;x<canvas.width;x =RESOLUTION_MIN){
80for(lety=RESOLUTION_MIN;y<canvas.height;y =RESOLUTION_MIN){
81//Convertxyco-ordstoarrayoffset.
82letn=y*canvas.width x;
83
84letfoundBodyPartNearby=false;
85
86//Let'scheckaroundagivenpixelifanyotherpixelswerebodylike.
87letyMin=y-SEARCH_OFFSET;
88yMin=yMin<0?0:yMin;
89
90letyMax=y SEARCH_OFFSET;
91yMax=yMax>canvas.height?canvas.height:yMax;
92
93letxMin=x-SEARCH_OFFSET;
94xMin=xMin<0?0:xMin;
95
96letxMax=x SEARCH_OFFSET;
97xMax=xMax>canvas.width?canvas.width:xMax;
98
99for(leti=xMin;i<xMax;i ){
100for(letj=yMin;j<yMax;j ){
101
102letoffset=j*canvas.width i;
103//Ifanyofthepixelsinthesquareweareanalysinghasabody
104//part,markascontaminated.
105if(segmentation.data[offset]!==0){
106foundBodyPartNearby=true;
107break;
108}
109}
110}
111
112//Updatepatchifpatchwasclean.
113if(!foundBodyPartNearby){
114for(leti=xMin;i<xMax;i ){
115for(letj=yMin;j<yMax;j ){
116//Convertxyco-ordstoarrayoffset.
117letoffset=j*canvas.width i;
118
119
120data[offset*4]=dataL[offset*4];
121data[offset*4 1]=dataL[offset*4 1];
122data[offset*4 2]=dataL[offset*4 2];
123data[offset*4 3]=255;
124}
125}
126}else{
127if(DEBUG){
128for(leti=xMin;i<xMax;i ){
129for(letj=yMin;j<yMax;j ){
130//Convertxyco-ordstoarrayoffset.
131letoffset=j*canvas.width i;
132
133
134data[offset*4]=255;
135data[offset*4 1]=0;
136data[offset*4 2]=0;
137data[offset*4 3]=255;
138}
139}
140}
141}
142
143
144}
145}
146ctx.putImageData(imageData,0,0);
147}
148
149//Let'sloadthemodelwithourparametersdefinedabove.
150//Beforewecanusebodypixclasswemustwaitforittofinish
151//loading.MachineLearningmodelscanbelargeandtakeamomentto
152//geteverythingneededtorun.
153varmodelHasLoaded=false;
154varmodel=undefined;
155
156model=bodyPix.load(bodyPixProperties).then(function(loadedModel){
157model=loadedModel;
158modelHasLoaded=true;
159//Showdemosectionnowmodelisreadytouse.
160demosSection.classList.remove('invisible');
161});
162
163/********************************************************************
164//Continuouslygrabimagefromwebcamstreamandclassifyit.
165********************************************************************/
166
167varpreviousSegmentationComplete=true;
168
169//Checkifwebcamaccessissupported.
170functionhasGetUserMedia(){
171return!!(navigator.mediaDevices&&
172navigator.mediaDevices.getUserMedia);
173}
174
175//Thisfunctionwillrepeatidlycallitselfwhenthebrowserisreadytoprocess
176//thenextframefromwebcam.
177functionpredictWebcam(){
178if(previousSegmentationComplete){
179//Copythevideoframefromwebcamtoatemporycanvasinmemoryonly(notintheDOM).
180videoRenderCanvasCtx.drawImage(video,0,0);
181previousSegmentationComplete=false;
182//Nowclassifythecanvasimagewehaveavailable.
183model.segmentPerson(videoRenderCanvas,segmentationProperties).then(function(segmentation){
184processSegmentation(webcamCanvas,segmentation);
185previousSegmentationComplete=true;
186});
187}
188
189//Callthisfunctionagaintokeeppredictingwhenthebrowserisready.
190window.requestAnimationFrame(predictWebcam);
191}
192
193//Enablethelivewebcamviewandstartclassification.
194functionenableCam(event){
195if(!modelHasLoaded){
196return;
197}
198
199//Hidethebutton.
200event.target.classList.add('removed');
201
202//getUsermediaparameters.
203constconstraints={
204video:true
205};
206
207//Activatethewebcamstream.
208navigator.mediaDevices.getUserMedia(constraints).then(function(stream){
209video.addEventListener('loadedmetadata',function(){
210//Updatewidthsandheightsoncevideoissuccessfullyplayedotherwise
211//itwillhavewidthandheightofzeroinitiallycausingclassification
212//tofail.
213webcamCanvas.width=video.videoWidth;
214webcamCanvas.height=video.videoHeight;
215videoRenderCanvas.width=video.videoWidth;
216videoRenderCanvas.height=video.videoHeight;
217letwebcamCanvasCtx=webcamCanvas.getContext('2d');
218webcamCanvasCtx.drawImage(video,0,0);
219});
220
221video.srcObject=stream;
222
223video.addEventListener('loadeddata',predictWebcam);
224});
225}
226
227//Wewillcreateatemporycanvastorendertostoreframesfrom
228//thewebcamstreamforclassification.
229varvideoRenderCanvas=document.createElement('canvas');
230varvideoRenderCanvasCtx=videoRenderCanvas.getContext('2d');
231
232//LetscreateacanvastorenderourfindingstotheDOM.
233varwebcamCanvas=document.createElement('canvas');
234webcamCanvas.setAttribute('class','overlay');
235liveView.appendChild(webcamCanvas);
236
237//Ifwebcamsupported,addeventlistenertobuttonforwhenuser
238//wantstoactivateit.
239if(hasGetUserMedia()){
240constenableWebcamButton=document.getElementById('webcamButton');
241enableWebcamButton.addEventListener('click',enableCam);
242}else{
243console.warn('getUserMedia()isnotsupportedbyyourbrowser');
244}
CSS：
1/**
2*@license
3*Copyright2018GoogleLLC.AllRightsReserved.
4*LicensedundertheApacheLicense,Version2.0(the"License");
5*youmaynotusethisfileexceptincompliancewiththeLicense.
6*YoumayobtainacopyoftheLicenseat
7*
8*http://www.apache.org/licenses/LICENSE-2.0
9*
10*Unlessrequiredbyapplicablelaworagreedtoinwriting,software
11*distributedundertheLicenseisdistributedonan"ASIS"BASIS,
12*WITHOUTWARRANTIESORCONDITIONSOFANYKIND,eitherexpressorimplied.
13*SeetheLicenseforthespecificlanguagegoverningpermissionsand
14*limitationsundertheLicense.
15*=============================================================================
16*/
17
18
19
20
21/******************************************************
22*StylesheetbyJasonMayes2020.
23*
24*GetlatestcodeonmyGithub:
25*https://github.com/jasonmayes/Real-Time-Person-Removal
26*Gotquestions?Reachouttomeonsocial:
27*Twitter:@jason_mayes
28*LinkedIn:https://www.linkedin.com/in/creativetech
29*****************************************************/
30
31
32
33
34body{
35font-family:helvetica,arial,sans-serif;
36margin:2em;
37color:#3D3D3D;
38}
39
40
41
42
43h1{
44font-style:italic;
45color:#FF6F00;
46}
47
48
49
50
51h2{
52clear:both;
53}
54
55
56
57
58em{
59font-weight:bold;
60}
61
62
63
64
65video{
66clear:both;
67display:block;
68}
69
70
71
72
73section{
74opacity:1;
75transition:opacity500msease-in-out;
76}
77
78
79
80
81header,footer{
82clear:both;
83}
84
85
86
87
88button{
89z-index:1000;
90position:relative;
91}
92
93
94
95
96.removed{
97display:none;
98}
99
100
101
102
103.invisible{
104opacity:0.2;
105}
106
107
108
109
110.note{
111font-style:italic;
112font-size:130%;
113}
114
115
116
117
118.webcam{
119position:relative;
120}
121
122
123
124
125.webcam,.classifyOnClick{
126position:relative;
127float:left;
128width:48%;
129margin:2%1%;
130cursor:pointer;
131}
132
133
134
135
136.webcamp,.classifyOnClickp{
137position:absolute;
138padding:5px;
139background-color:rgba(255,111,0,0.85);
140color:#FFF;
141border:1pxdashedrgba(255,255,255,0.7);
142z-index:2;
143font-size:12px;
144}
145
146
147
148
149.highlighter{
150background:rgba(0,255,0,0.25);
151border:1pxdashed#fff;
152z-index:1;
153position:absolute;
154}
155
156
157
158
159.classifyOnClick{
160z-index:0;
161position:relative;
162}
163
164
165
166
167.classifyOnClickcanvas,.webcamcanvas.overlay{
168opacity:1;
169
170top:0;
171left:0;
172z-index:2;
173}
174
175
176
177
178#liveView{
179transform-origi
Html:
1<!DOCTYPEhtml>
2<htmllang="en">
3<head>
4<title>DisappearingPeopleProject</title>
5<metacharset="utf-8">
6<metahttp-equiv="X-UA-Compatible"content="IE=edge">
7<metaname="viewport"content="width=device-width,initial-scale=1">
8<metaname="author"content="JasonMayes">
9
10
11
12
13<!--Importthewebpage'sstylesheet-->
14<linkrel="stylesheet"href="/style.css">
15
16
17
18
19<!--ImportTensorFlow.jslibrary-->
20<scriptsrc="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs/dist/tf.min.js"type="text/javascript"></script>
21</head>
22<body>
23<h1>DisappearingPeopleProject</h1>
24
25<headerclass="note">
26<h2>RemovingpeoplefromcomplexbackgroundsinrealtimeusingTensorFlow.js</h2>
27</header>
28
29
30
31
32<h2>Howtouse</h2>
33<p>Pleasewaitforthemodeltoloadbeforetryingthedemosbelowatwhichpointtheywillbecomevisiblewhenreadytouse.</p>
34<p>Hereisavideoofwhatyoucanexpecttoachieveusingmycustomalgorithm.Thetopistheactualfootage,thebottomvideoiswiththerealtimeremovalofpeopleworkinginJavaScript!</p>
35<iframewidth="540"height="812"src="https://www.youtube.com/embed/0LqEuc32uTc?controls=0&autoplay=1"frameborder="0"allow="accelerometer;autoplay;encrypted-media;gyroscope;picture-in-picture"allowfullscreen></iframe>
36
37<sectionid="demos"class="invisible">
38
39
40
41
42<h2>Demo:Webcamliveremoval</h2>
43<p>Trythisoutusingyourwebcam.Standafewfeetawayfromyourwebcamandstartwalkingaround...Watchasyouslowlydisappearinthebottompreview.</p>
44
45<divid="liveView"class="webcam">
46<buttonid="webcamButton">EnableWebcam</button>
47<videoid="webcam"autoplay></video>
48</div>
49</section>
50
51
52
53
54
55<!--IncludetheGlitchbuttontoshowwhatthewebpageisaboutand
56tomakeiteasierforfolkstoviewsourceandremix-->
57<divclass="glitchButton"style="position:fixed;top:20px;right:20px;"></div>
58<scriptsrc="https://button.glitch.me/button.js"></script>
59
60<!--Loadthebodypixmodeltorecognizebodypartsinimages-->
61<scriptsrc="https://cdn.jsdelivr.net/npm/@tensorflow-models/body-pix@2.0"></script>
62
63<!--Importthepage'sJavaScripttodosomestuff-->
64<scriptsrc="/script.js"defer></script>
65</bod
66
实时演示
你也可以在自己的Web浏览器中根据自己的喜好试着复现一下：
Codepen.io：https://codepen.io/jasonmayes/pen/GRJqgma
Glitch.com：https://glitch.com/~disappearing-people
等待模型加载完成，然后就可以使用了。
这是使用作者自定义算法实现的视频。上半部分是实际镜头，底部是用JavaScript实时删除人物的视频。
用你自己的网络摄像头试一下，要距离摄像头几英尺远，然后来回走动，在底部预览中你会慢慢从画面中消失。赶快试试吧，使用效果别忘了留言和大家一起分享哦！
,
