生而为人

best practise

新建cluster注意事项：

指定独立的VNet，保证可以做灵活的网络管控
对于节点多的cluster，为了保证Ambari监控性能，最好申请独立的DB，并且提高其等级，机器越多建议提高的等级越多。

NuGet

Posted on 2026-08-07 In microsoft

Package Manager Console

参考资料：

Package Manager Console (PowerShell)
1. 如何使用PMC
2. 命令list
PowerShell reference
1. 命令明细

Untitled

Posted on 2026-08-07

d:\project\APGold\autopilotservice\Global\VirtualEnvironments\AdsBI\AdsMz-Test-MW1-MSNMediation>sd edit deployment.int
deployment.int - file(s) not on client.

直接编辑

add/edit
安装codeflow 提交pr
\codeflow.redmond.corp.microsoft.com\public\cf2Launcher.cmd

codeflow作用

ve作用：
pe作用：

之前

D:\project\APGold\autopilotservice\Global\VirtualEnvironments\AdsBI\AdsMz-Test-MW1-MSNMediation

D:\project\APGold\autopilotservice\MW1\AdsMz-Prod-MW1

A ServiceGroup is mandatory for every machine function

VE中存储的可以理解为某个项目的global config，所有的PE公用这部分config
设计目的是在同一项目部署多个PE时，避免多次发布，只需要更新VE，相关PE就会生效

https://msazure.visualstudio.com

D:\project\APGold\autopilotservice\Global\VirtualEnvironments\AdsBI\AdsMz-Prod-MW1-MSNMediation

07/26/21 16:54:01.841,error processing AdsMz-Test-MW1: Rollout 'AdsBI-AdsMz-Test-MW1-MzOrchestration-VE.6799404_17553440741849145769_0.Rollout_AdsBI_Streaming_CFR' cannot be kicked due to status: InRollout - Rollout in progress..Rollout 'AdsBI-AdsMz-Test-MW1-MSNMediation-VE.6827694_4064931696698155710_0.Rollout_AdsBI_Streaming_CFR' cannot be kicked due to status: OtherRollout - Waiting on other rollout on MF AdsBI_Streaming_CFR in progress: AdsBI_Streaming_CFR.AdvertiserAggs_94d0ef91_90546_1_15-0-897+94d0ef91_CL6670088_5920165827906938206_0.AdsBI-AdsMz-Test-MW1-AdvertiserAggs-VE.csv:AdsBI_Streaming_CFR.AgoraAggs_76700988_85934_1_16-0-173+76700988_CL6670088_4056715873563314682_0.AdsBI-AdsMz-Test-MW1-AgoraAggs-VE.csv:AdsBI_Streaming_CFR.CACFR_9a63868e_90143_2_1-0-2408+9a63868e_CL6670088_11167218336511992544_0.AdsBI-AdsMz-Test-MW1-CACFR-VE.csv:AdsBI_Streaming_CFR.CFR_2d2cea68_91067_1_17-0-6662+2d2cea68_CL6670088_18415503406869780426_0.AdsBI-AdsMz-Test-MW1-CFR-VE.csv:AdsBI_Streaming_CFR.KpiAggs_fdd5f92d_90180_1_17-0-433+fdd5f92d_CL6670088_6626197140932378707_0.AdsBI-AdsMz-Test-MW1-KPIAggs-VE.csv:AdsBI_Streaming_CFR.Monetization_de0db608_91071_1_1-0-2808+de0db608_CL6670088_351842135959690630_0.AdsBI-AdsMz-Test-MW1-Monetization-VE.csv:AdsBI_Streaming_CFR.Orchestration_93eb645e_91078_1_16-0-557+93eb645e_CL6799404_17553440741849145769_0.AdsBI-AdsMz-Test-MW1-MzOrchestration-VE.csv:AdsBI_Streaming_CFR.PublisherAggs_55bd6dd9_90534_1_17-0-474+55bd6dd9_CL6670088_15361642287162420254_0.AdsBI-AdsMz-Test-MW1-PublisherAggs-VE.csv:AdsBI_Streaming_CFR.Mediation_865d3b39_90170_1_merge_20210629_-1_CL6670088_3189803787641037971_0.AdsBI-AdsMz-Test-MW1-Mediation-VE.csv:AdsBI_Streaming_CFR.MSNMediation_4a63e32e_91077_2_0-1-28+4a63e32e33_CL6827694_4798040603905411799_0.AdsBI-AdsMz-Test-MW1-MSNMediation-VE.csv:.OM.Autopilot-AutopilotClient-VE.csv..; .. Some rollouts not triggered.

Deployment: 6827694_4798040603905411799 (this is NOT the latest deployment.ini change number)
Downloaded 89 files, 21510455 bytes, compressed size: 8267353 bytes.
Failed to build services: Failed to build service 'SAMHourlyS2B.MSNMediation_4a63e32e_91077_2_0-1-28+4a63e32e33': Proxy error: not all data was received (DP: 25.68.114.53; by MW01NAP0000036C); ;  ;Log:e:GetStreamChunkMapWithRetry:stream does not exist [response from server] : GetStreamChunkMap('stream://AdsBI-AdsMz-Test-MW1-MSNMediation-VE/app/ServiceMaps/MSNMediationServiceMap.ini'; 'MSNMediation_4a63e32e_91077_2_0-1-28+4a63e32e33') failed ;Log:w:ApDynamicStorage::StorageConnectionEx::DownloadBufferFromStreamWithRetry:Stream (stream://AdsBI-AdsMz-Test-MW1-MSNMediation-VE/app/ServiceMaps/MSNMediationServiceMap.ini _ MSNMediation_4a63e32e_91077_2_0-1-28+4a63e32e33) not found in storage ;
Majority of the building errors are temporary and will be fixed automatically. 
If there is no change or progress in the app deployment log after 15 minutes, contact apswat for production environments and aptalk for non-production environment.
Look in App Deployment log for more info

Here are common error messages and possible fixes:
     Msg: Error 53: The network path was not found.: ac: 0
     Fix: Try adding REDMOND@ to the build path.

     Msg: Msg: EDP010385: APSEQREAD::GetDataPointer error 2
     Fix: Try adding REDMOND@ to the build path.

     Msg: Proxy reported error: EDP010196: ApckBuilder Error 3: The system cannot find the path specified.
     Fix: Look for missing directories under your build drop.

For more troubleshooting info, please refer to App Deployment Troubleshooting

Deployment: 6827694_9979618793807238839_0 (this is NOT the latest deployment.ini change number)
Downloaded [unknown] files, [unknown] bytes.
Failed to import version '6827694_9979618793807238839' of image for environment 'AdsBI-AdsMz-Test-MW1-MSNMediation-VE': Error downloading image from remote storage (stream 'stream://AdsBI-AdsMz-Test-MW1-MSNMediation-VE/app/image.ini@6827694_9979618793807238839')
Majority of the building errors are temporary and will be fixed automatically. 
If there is no change or progress in the app deployment log after 15 minutes, contact apswat for production environments and aptalk for non-production environment.
Look in App Deployment log for more info

Here are common error messages and possible fixes:
     Msg: Error 53: The network path was not found.: ac: 0
     Fix: Try adding REDMOND@ to the build path.

     Msg: Msg: EDP010385: APSEQREAD::GetDataPointer error 2
     Fix: Try adding REDMOND@ to the build path.

     Msg: Proxy reported error: EDP010196: ApckBuilder Error 3: The system cannot find the path specified.
     Fix: Look for missing directories under your build drop.

For more troubleshooting info, please refer to App Deployment Troubleshooting

Severity Code Description Project File Line Suppression State
Error MSB3680 The source file “D:\project\Ads.BI.MSNMediation\private\src\Batch/Autopilot/ExternalConfigs/Microsoft.BingAds.BI.APDeploy.exe.config” does not exist. Microsoft.BI.MSNMediation.HourlyS2B.Drop D:\project\Ads.BI.MSNMediation\private\src\Batch\build\targets\CreateDrop.targets 47

、

Severity Code Description Project File Line Suppression State
Error MSB3030 Could not copy the file “D:\project\Ads.BI.MSNMediation\private\src\Batch/SAMHourlyS2B/Microsoft.BI.MSNMediation.HourlyS2B.Drop/bin\Debug\net472**” because it was not found. Microsoft.BI.MSNMediation.HourlyS2B.Drop D:\project\Ads.BI.MSNMediation\private\src\Batch\build\targets\CreateDrop.targets 47

c:\Users\jingqicao.nuget\packages\microsoft.bingads.bi.apdeploy\10.5.2

在sdk之外做的nuget package改动，比如删掉某个包

resolved Show files that have been merged but not submitted 跟执行目录无关
retype Reappraise the file type for files on the client
revert Discard changes from an opened file
review List and track changelists (for the review daemon)

sd review 如何只搜索某个人的review

FAREAST\jingqicao

sd opened

sd submit -c

Untitled

Posted on 2026-08-07

azure cosmos命令

Untitled

Posted on 2026-08-07

Azure HDInsight documentation

Update Azure storage account access keys in HDInsight cluster 当更新了remote key时，需要执行相关脚本，让节点更新。Azure Blob Storage fails to authenticate: “Make sure the value of Authorization header is formed correctly including the signature”

Azure Samples

Virtual network

记录

Posted on 2026-08-07 In microsoft

工具准备

scope插件安装

https://mscosmos.visualstudio.com/CosmosWiki/_wiki/wikis/Cosmos.wiki/66/SCOPE-Tutorial

时区

UTC时间
美国时间：UTC - 7
北京时间：UTC + 8

服务器时间： UTC

xts log中打印的时间：UTC

xts log构建时间(xts服务器时间)：美国时间

Untitled

Posted on 2026-08-07

puid
gid 999+工号
company simple： MSFT

ESPP
R开头的一串数字

HRWeb
信息维护，会影响摩根账户，需要全部都是英文，包括地址
https://myaccount.microsoft.com/?ref=MeControl

中意人寿
family

143cc3b8f01f
https://apkpure.com

部署任务

Posted on 2026-08-07 In microsoft

Auzre DevOps任务定时调度

步骤：Onboard AP App Deployment using AzDeployer (Stratus)

Prepare workflow folder, master config and workflow config.
Prepare APDrop.
Create APGold virtual environment.
Setup corresponding Azure Dev Ops CD pipeline. Add Azure DevOps release pipeline

##[warning]C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\amd64\Microsoft.Common.CurrentVersion.targets(2203,5): Warning MSB3245: Could not resolve this reference. Could not locate the assembly "Google.Protobuf, Version=3.6.1.0, Culture=neutral, PublicKeyToken=a7d26565bac4d604, processorArchitecture=MSIL". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors.


externalFeedCredentials: 'nuget-msazure-oldbond, nuget-msblox-azuregenevamonitoring, nuget-mscosmos-cosmosprod, nuget-mscosmos-cosmostest, nuget-msdata-bigdata, nuget-ossmsft-oss_all, nuget-trill-trill'

2021-08-02 06:33:05 ThreadId:   69	PauseAndResumeWorkflows() Process: MSNBI_S2BHourly_0 UserCommand: Run
2021-08-02 06:33:05 ThreadId:   79	Getting information about event AdsBI_MSNBI_S2B_Hourly_Done
2021-08-02 06:33:05 ThreadId:   88	Getting information about event AdsBI_MSNBI_S2B_Hourly_Done
2021-08-02 06:33:05 ThreadId:   83	Getting information about event AdsBI_MSNBI_S2B_Hourly_Done
2021-08-02 06:33:05 ThreadId:   83	Got SQL exception: System.Data.SqlClient.SqlException (0x80131904): -999999:3:2--prc_ProcessStateGetNextDelta-212--Violation of PRIMARY KEY constraint 'PK_PROCESSPRESTATE'. Cannot insert duplicate key in object 'dbo.ProcessPrestate'. The duplicate key value is (-1931531102, 20210731 1100).
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader()
   at Microsoft.AdCenter.WorkflowHost.CommunicationWorkflow.<>c__DisplayClass32_0.<GetLastParam>b__1(SqlConnection sqlConnection)
   at Microsoft.AdCenter.WorkflowHost.SqlConnectionHelper.RunQuery(Action`1 command)
ClientConnectionId:0fd2a26b-bb72-4bbd-baec-a16499dff840
Error Number:50000,State:1,Class:16
ClientConnectionId before routing:bfc0cfb4-53ac-4a8f-a985-44e2e512efe2
Routing Destination:b9d5d60a293f.tr147.westus2-a.worker.database.windows.net,11020
connection string: Server=adsdwctest.database.windows.net;Database=DWC_DB;User ID=AdsDataSI_Execution;Password=AnotherPassword8!

2021-08-02T08:28:02.5629452Z          Checking compatibility for System.Security.Cryptography.Primitives 4.3.0 with .NETFramework,Version=v4.7.2.
2021-08-02T08:28:02.5630121Z          All packages and projects are compatible with .NETFramework,Version=v4.7.2.
2021-08-02T08:28:02.6040861Z          Committing restore...
2021-08-02T08:28:02.6041989Z          Generating MSBuild file D:\a\1\s\private\src\Batch\SAMHourlyS2B\Microsoft.BI.MSNMediation.HourlyS2B.Drop\obj\Microsoft.BI.MSNMediation.HourlyS2B.Drop.csproj.nuget.g.props.
2021-08-02T08:28:02.6048518Z          Generating MSBuild file D:\a\1\s\private\src\Batch\SAMHourlyS2B\Microsoft.BI.MSNMediation.HourlyS2B.Drop\obj\Microsoft.BI.MSNMediation.HourlyS2B.Drop.csproj.nuget.g.targets.
2021-08-02T08:28:02.6053133Z          Writing assets file to disk. Path: D:\a\1\s\private\src\Batch\SAMHourlyS2B\Microsoft.BI.MSNMediation.HourlyS2B.Drop\obj\project.assets.json
2021-08-02T08:28:02.6174412Z          Writing cache file to disk. Path: D:\a\1\s\private\src\Batch\SAMHourlyS2B\Microsoft.BI.MSNMediation.HourlyS2B.Drop\obj\project.nuget.cache
2021-08-02T08:28:02.6187683Z          Persisting dg to D:\a\1\s\private\src\Batch\SAMHourlyS2B\Microsoft.BI.MSNMediation.HourlyS2B.Drop\obj\Microsoft.BI.MSNMediation.HourlyS2B.Drop.csproj.nuget.dgspec.json
2021-08-02T08:28:02.6200559Z          Failed to restore D:\a\1\s\private\src\Batch\SAMHourlyS2B\Microsoft.BI.MSNMediation.HourlyS2B.Drop\Microsoft.BI.MSNMediation.HourlyS2B.Drop.csproj (in 17.33 sec).
2021-08-02T08:28:02.6260383Z          
2021-08-02T08:28:02.6264281Z          NuGet Config files used:
2021-08-02T08:28:02.6264975Z              D:\a\1\Nuget\tempNuGet_22994537.config
2021-08-02T08:28:02.6265449Z          
2021-08-02T08:28:02.6266040Z          Feeds used:
2021-08-02T08:28:02.6266664Z              https://msasg.pkgs.visualstudio.com/Shared%20Data/_packaging/Ads.BI.SubjectArea.Upstreams/nuget/v3/index.json
2021-08-02T08:28:02.6267321Z          
2021-08-02T08:28:02.6267938Z          Installed:
2021-08-02T08:28:02.6281713Z              60 package(s) to D:\a\1\s\private\src\Batch\SAMHourlyS2B\Microsoft.BI.MSNMediation.HourlyS2B.Drop\Microsoft.BI.MSNMediation.HourlyS2B.Drop.csproj
2021-08-02T08:28:02.6297867Z        Done executing task "RestoreTask" -- FAILED.
2021-08-02T08:28:02.6302830Z      1>Done building target "Restore" in project "dirs.proj" -- FAILED.
2021-08-02T08:28:02.6303710Z      1>Done Building Project "D:\a\1\s\private\src\Batch\dirs.proj" (Restore target(s)) -- FAILED.
2021-08-02T08:28:02.6405509Z 
2021-08-02T08:28:02.6547930Z Build FAILED.
2021-08-02T08:28:02.6562009Z 
2021-08-02T08:28:02.6580770Z        "D:\a\1\s\private\src\Batch\dirs.proj" (Restore target) (1) ->
2021-08-02T08:28:02.6589059Z        (Restore target) -> 
2021-08-02T08:28:02.6596250Z          D:\a\1\s\private\src\Batch\SAMHourlyS2B\Microsoft.BI.MSNMediation.HourlyS2B.Drop\Microsoft.BI.MSNMediation.HourlyS2B.Drop.csproj : error NU1605: Detected package downgrade: Microsoft.BI.Common from 4.21.0 to 4.5.0. Reference the package directly from the project to select a different version.  [D:\a\1\s\private\src\Batch\dirs.proj]
2021-08-02T08:28:02.6599483Z        D:\a\1\s\private\src\Batch\SAMHourlyS2B\Microsoft.BI.MSNMediation.HourlyS2B.Drop\Microsoft.BI.MSNMediation.HourlyS2B.Drop.csproj : error NU1605:  Microsoft.BI.MSNMediation.HourlyS2B.Drop -> Ads.BI.StreamingToBatch 2.2.0 -> Microsoft.BI.Common (>= 4.21.0)  [D:\a\1\s\private\src\Batch\dirs.proj]
2021-08-02T08:28:02.6602014Z        D:\a\1\s\private\src\Batch\SAMHourlyS2B\Microsoft.BI.MSNMediation.HourlyS2B.Drop\Microsoft.BI.MSNMediation.HourlyS2B.Drop.csproj : error NU1605:  Microsoft.BI.MSNMediation.HourlyS2B.Drop -> Microsoft.BI.Common (>= 4.5.0) [D:\a\1\s\private\src\Batch\dirs.proj]
2021-08-02T08:28:02.6603542Z          D:\a\1\s\private\src\Batch\SAMHourlyS2B\Microsoft.BI.MSNMediation.HourlyS2B.Drop\Microsoft.BI.MSNMediation.HourlyS2B.Drop.csproj : error NU1605: Detected package downgrade: Microsoft.BI.Common from 4.12.0 to 4.5.0. Reference the package directly from the project to select a different version.  [D:\a\1\s\private\src\Batch\dirs.proj]
2021-08-02T08:28:02.6605044Z        D:\a\1\s\private\src\Batch\SAMHourlyS2B\Microsoft.BI.MSNMediation.HourlyS2B.Drop\Microsoft.BI.MSNMediation.HourlyS2B.Drop.csproj : error NU1605:  Microsoft.BI.MSNMediation.HourlyS2B.Drop -> Ads.BI.Orchestration.Workflows 0.1.0 -> Microsoft.BI.Common (>= 4.12.0)  [D:\a\1\s\private\src\Batch\dirs.proj]
2021-08-02T08:28:02.6606611Z        D:\a\1\s\private\src\Batch\SAMHourlyS2B\Microsoft.BI.MSNMediation.HourlyS2B.Drop\Microsoft.BI.MSNMediation.HourlyS2B.Drop.csproj : error NU1605:  Microsoft.BI.MSNMediation.HourlyS2B.Drop -> Microsoft.BI.Common (>= 4.5.0) [D:\a\1\s\private\src\Batch\dirs.proj]
2021-08-02T08:28:02.6608307Z          D:\a\1\s\private\src\Batch\SAMHourlyS2B\Microsoft.BI.MSNMediation.HourlyS2B.Drop\Microsoft.BI.MSNMediation.HourlyS2B.Drop.csproj : error NU1605: Detected package downgrade: Microsoft.Bingads.Dwc.Tools from 2.2.0 to 2.0.5102033.20. Reference the package directly from the project to select a different version.  [D:\a\1\s\private\src\Batch\dirs.proj]
2021-08-02T08:28:02.6609947Z        D:\a\1\s\private\src\Batch\SAMHourlyS2B\Microsoft.BI.MSNMediation.HourlyS2B.Drop\Microsoft.BI.MSNMediation.HourlyS2B.Drop.csproj : error NU1605:  Microsoft.BI.MSNMediation.HourlyS2B.Drop -> Ads.BI.Orchestration.Workflows 0.1.0 -> Microsoft.Bingads.Dwc.Tools (>= 2.2.0)  [D:\a\1\s\private\src\Batch\dirs.proj]
2021-08-02T08:28:02.6612139Z        D:\a\1\s\private\src\Batch\SAMHourlyS2B\Microsoft.BI.MSNMediation.HourlyS2B.Drop\Microsoft.BI.MSNMediation.HourlyS2B.Drop.csproj : error NU1605:  Microsoft.BI.MSNMediation.HourlyS2B.Drop -> Microsoft.BingAds.Dwc.Tools (>= 2.0.5102033.20) [D:\a\1\s\private\src\Batch\dirs.proj]
2021-08-02T08:28:02.6612903Z 
2021-08-02T08:28:02.6613320Z     0 Warning(s)
2021-08-02T08:28:02.6613727Z     3 Error(s)
2021-08-02T08:28:02.6613981Z 
2021-08-02T08:28:02.6614474Z Time Elapsed 00:00:19.93
2021-08-02T08:28:02.7009246Z ##[error]Error: The process 'C:\Program Files\dotnet\dotnet.exe' failed with exit code 1
2021-08-02T08:28:02.7020201Z ##[error]Packages failed to restore









可知，package引入如果指定了低层级的版本，会强制上层package统一使用，类似java的原理。



    <PackageReference Include="Ads.BI.BinplacingTools">
      <Version>3.0.0</Version>
    </PackageReference>
    <PackageReference Include="Ads.BI.PipelineApps.Egress">
      <Version>6.1.0</Version>
    </PackageReference>
    <PackageReference Include="Microsoft.AdCenter.ExecuteSSISPackage">
      <Version>1.1.5102398-pacman</Version>
    </PackageReference>
    <PackageReference Include="Microsoft.Azure.KeyVault.Core">
      <Version>1.0.0</Version>
    </PackageReference>
    <PackageReference Include="Microsoft.BingAds.Dwc.Engine.Library">
      <Version>2.1.0</Version>
    </PackageReference>
    <PackageReference Include="Microsoft.Search.Autopilot">
      <Version>1.0.0</Version>
    </PackageReference>
    <PackageReference Include="WindowsAzure.Storage">
      <Version>7.1.2</Version>
    </PackageReference>

ads Data live-site tracking v2

Deployment process for FASTBI streaming pipeline - V2

布隆过滤器

Posted on 2026-08-07 In 算法

布隆过滤器(Bloom Filter)详解

生而为人

HDInsight

guide line

best practise

新建cluster注意事项：

NuGet

guide line

Restore packages

Package Manager Console

Untitled

Untitled

Untitled

Untitled

记录

工具准备

Untitled

部署任务

Auzre DevOps任务定时调度

布隆过滤器